What Is Speculative Decoding and Why Does It Matter?

Language models generate text one token at a time, and each token requires a full pass through the model. For a large model, that is slow and expensive, because every single token means another heavy computation. Speculative decoding is a clever way to speed this up without changing what the model produces.

The idea is to let a small, fast model guess several tokens ahead, and then have the large model check all of those guesses at once. Verifying a batch of tokens in one pass is much faster than generating them one by one, so the large model effectively produces several tokens per step. This guide explains how it works and why it matters.

We will cover the definition, how it works, how it compares to standard decoding, and when to use it.

What is speculative decoding?

Speculative decoding is an inference optimization that pairs a large target model with a small, fast draft model. The draft model proposes several upcoming tokens quickly, and the large model verifies them all in a single forward pass, accepting the ones it agrees with and correcting the first one it does not. The result is the same output the large model would have produced, generated faster.

Think of it like a junior writer drafting a sentence and a senior editor checking it in one quick read. The editor keeps the parts that are right and fixes the first mistake, which is far faster than writing every word from scratch. Speculative decoding pairs a fast drafter with a careful verifier the same way.

The key idea is draft-then-verify. Because checking several proposed tokens together is cheaper than generating them individually, the expensive model does less sequential work while producing exactly the output it otherwise would.

How does speculative decoding work?

A small model drafts tokens; the large model verifies them in one pass.

• Draft ahead. The small model quickly proposes several next tokens.
• Verify together. The large model checks them all in one forward pass.
• Accept matches. Keep the leading tokens the large model agrees with.
• Correct the rest. Fix the first disagreement and discard what follows.
• Repeat. Draft the next batch from where verification left off.

The speedup comes because the large model verifies a batch of tokens in roughly the time it would take to generate one. When the small model's guesses are often right, many tokens are accepted per step, so generation accelerates. Crucially, the verification step guarantees the final output matches what the large model alone would produce.

Speculative vs. standard decoding

Standard decoding generates one token at a time, each requiring a full pass through the large model. It is simple and exact, but inherently sequential, so the large model's size directly limits speed, and every token pays the full cost.

Speculative decoding keeps the exact same output but reduces the number of expensive sequential steps by verifying drafted tokens in batches. The tradeoff is the added complexity of running a second draft model and the fact that the gain depends on how often the draft is right. When the draft model is well matched, the speedup can be substantial at no cost to quality.

Why does speculative decoding matter?

• Speed. It generates text faster from large models.
• Same output. It does not change what the model produces.
• Lower cost. Faster generation means cheaper inference.
• Better latency. Responses come back more quickly.
• Efficiency. It gets more out of expensive hardware.

What are the tradeoffs?

• Extra model. It requires running a separate draft model.
• Variable gain. The speedup depends on draft accuracy.
• Matching. The draft model must suit the target model well.
• Complexity. The serving setup is more involved.

When should you use speculative decoding?

Use speculative decoding when you are serving a large language model and want faster generation and lower latency without sacrificing output quality. It is especially valuable for interactive applications, where response speed matters, and at scale, where faster inference directly reduces cost.

If you already use a small model, or latency is not a concern, the added complexity may not be worth it. But when you need the quality of a large model with better speed, speculative decoding is one of the most effective ways to get it, since it changes nothing about the output.

Key points about speculative decoding

• Speculative decoding speeds up generation without changing output.
• A small model drafts tokens; a large model verifies them.
• Verifying a batch is faster than generating token by token.
• Accepted tokens are kept; the first wrong one is corrected.
• The gain depends on how often the draft is right.
• It lowers inference latency and cost.

Conclusion

Speculative decoding speeds up large language model generation by having a small, fast model draft several tokens ahead and a large model verify them in a single step. Because batch verification is faster than token-by-token generation, text comes out quicker, and the verification guarantees the output is exactly what the large model would have produced. It is a key optimization for faster, cheaper, lower-latency inference.

TrueFoundry helps teams serve models efficiently on their own infrastructure, including inference optimizations like speculative decoding that cut latency and cost without touching quality. When you need large-model answers delivered fast, speculative decoding is a powerful lever.