Speculative Decoding: Why Running Two LLMs Can Be Faster Than Running One

One of the fastest ways to run a large LLM is to run a second, smaller model next to it. The small model guesses several tokens ahead, the large model checks all the guesses in one forward pass, and if the guesses are good, one expensive pass advances generation by several tokens instead of one. The part that surprises most people: with the right acceptance algorithm this is not an approximation. The output distribution is exactly the target model's.

That sounds backwards at first. You have a 70B model that is already expensive to serve. Why would loading another 1B or 7B model make anything faster? More models should mean more compute, not less.

The answer comes from looking at inference as a systems problem. The bottleneck in autoregressive generation is not “the model needs too many FLOPs.” The bottleneck is that we run an expensive forward pass and get exactly one token out of it, over and over. Speculative decoding restructures that. Instead of asking the big model for one token at a time, we hand it five guesses and ask it to check all five at once. Transformers can evaluate multiple positions in parallel, so the checking step is far cheaper per token than the generating step.

Take the prompt “The capital of France is”. The model produces “Paris”, but internally it works one token at a time:

Copy
P(token_1 | prompt)
P(token_2 | prompt, token_1)
P(token_3 | prompt, token_1, token_2)

The dependency is x_t ~ P(x_t | x_1, ..., x_{t-1}). You cannot compute token t+1 before you have token t. If the model is producing “The quick brown fox jumps over the lazy dog”, you don't know fox until you've selected brown, and you don't know jumps until you've selected fox. GPUs are built for massive parallel work, and autoregressive decoding forces them into a serial loop. The original speculative decoding paper frames the problem exactly this way: generating K tokens normally takes K serial runs of the model.

Inference has two stages, and they behave nothing alike.

During prefill, every prompt token already exists. A 2,000-token prompt gives the GPU 2,000 positions it can process in parallel, so prefill exposes plenty of work.

During decode, each sequence contributes one new token per step. Forward pass, sample, append, repeat. The KV cache saves us from recomputing attention keys and values for the whole history, which matters enormously, but it doesn't remove the serial dependency between output tokens. So decode ends up limited by memory movement and poor GPU utilization rather than raw arithmetic capacity. Speculative decoding uses the spare compute that sits idle during this memory-bound decode phase. Everything else in this post builds on that observation.

If the context is “The capital of France is”, you do not need 70B parameters to predict “Paris”. If the context is for (int i = 0;, a small code model will confidently and correctly continue i < n; i++). “Thank you very much for your” is almost always followed by “help” or something close to it.

Language is full of easy local decisions. The large model earns its cost on the hard ones, but a lot of individual token transitions don't need its full capacity. Which suggests an idea: let a cheap model make the easy predictions, and only ask the expensive model to verify them. That's the whole intuition.

The target model is the one whose output you actually want, say Llama 70B. The draft model is much cheaper: a 1B or 3B sibling, or a model trained specifically to approximate the target.

Copy
                 ┌───────────────────┐
Prompt ─────────►│   Draft Model     │
                 │      small        │
                 └─────────┬─────────┘
                           ▼
                 [ t1 t2 t3 t4 t5 ]
                           │
                           ▼
                 ┌───────────────────┐
                 │   Target Model    │
                 │      large        │
                 └─────────┬─────────┘
                           │ verifies them
                           ▼
                 accept / reject

Say K = 4 and the context is “The quick”. The draft proposes brown fox hopped over. Instead of generating four tokens sequentially with the target, we feed the full speculative continuation “The quick brown fox hopped over” into the target once. Because of the causal mask, the target gets predictions for every speculative position in one batched pass.

Maybe the target agrees with brown and fox but disagrees at hopped. We keep brown fox, the target corrects the next token to jumped, and generation continues from there. One target invocation advanced us three tokens instead of one. That's the entire win.

The first instinct is usually “just compare the small model's tokens with what the big model would have picked.” That works as an intuition for greedy decoding, but it breaks for real sampling.

Suppose the target's distribution at some position is Paris 0.70, London 0.10, Lyon 0.08, Berlin 0.05, and the draft's is Paris 0.85, London 0.05, Lyon 0.03, Berlin 0.02. If we accept whatever the draft proposes, Paris now appears too often. We're no longer sampling from the target; we've quietly built an approximation.

The good news is we don't have to accept naively. The acceptance procedure can mathematically correct for the gap between the two distributions.

Call the target distribution p(x) and the draft distribution q(x). The draft samples a token x ~ q(x), and we need to decide whether to keep it as though it came from p. The classic rule accepts with probability:

Copy
A(x) = min(1, p(x)/q(x))

Two cases make the intuition click. If p(x) = 0.8 and q(x) = 0.4, the ratio is 2, so A(x) = 1 and we always accept. The draft is actually underproducing this token relative to the target, so no correction is needed. If p(x) = 0.2 and q(x) = 0.5, we accept only 40% of the time, because the draft proposes this token more often than the target would, and the rejections cancel that bias out.

On rejection we don't just resample from p. We sample from the residual distribution:

Copy
p'(x) ∝ max(0, p(x) − q(x))

which puts back exactly the probability mass the rejections removed. This is the clever part of the algorithm: the draft is allowed to have a different distribution because the verifier corrects the difference. The final tokens follow the target model's distribution exactly, which is the central result of Leviathan, Kalman, and Matias.

So when people call speculative decoding “lossless”, this is what they mean: it preserves the target's sampling distribution. That's more precise than claiming every run produces the identical token sequence, which sampling never guaranteed anyway.

This is the point that trips people up most. If token 4 depends on token 3, how can the target check them in parallel?

Because the draft already gave us candidate values for all of them. Given draft tokens d1, d2, d3, d4, the target evaluates p(d1 | x), p(d2 | x, d1), p(d3 | x, d1, d2), and p(d4 | x, d1, d2, d3) in a single invocation over the speculative block. We are not asking the target to discover four tokens simultaneously, which would violate the autoregressive dependency. We're saying: assume these are the continuation, and tell us what probability you assign to each. Once the candidates exist, this is ordinary masked parallel sequence computation, the same thing that makes prefill fast. Sequential generation becomes parallel verification, and GPUs are very good at the second problem.

Put rough numbers on it. One target decode step costs T_T and yields one token, so baseline cost per token is about T_T. A speculative round costs roughly K·T_D + T_V(K), where T_D is one draft step, K is the speculation length, and T_V(K) is the target's verification cost for the block. If a round commits τ tokens on average, then:

Copy
cost/token = (K·T_D + T_V(K)) / τ

and speculation wins when that number is below T_T. This one inequality explains basically every benchmark you'll ever see. You need a cheap drafter, high target-draft agreement, efficient verification, and enough accepted tokens to pay for the extra work. A second model is not automatically useful. It's useful only when the speculative work it adds costs less than the sequential work it removes.

Let α be the per-token acceptance probability and assume, for a simplified model, that it's constant across positions. Then the expected progress per round with speculation length K is:

Copy
τ = (1 − α^(K+1)) / (1 − α)

At α = 0.84 and K = 5, τ ≈ 4.05: one verification round advances you four tokens on average, which is excellent. At α = 0.60, τ ≈ 2.38, still useful but much less exciting. At α = 0.40, τ ≈ 1.66. At that point you've run the draft five times, done a verification pass, burned extra memory, and coordinated two models to advance fewer than two tokens on average. The optimization can easily stop paying for itself. Speculative decoding performance is fundamentally an acceptance-rate problem.

If five speculative tokens help, wouldn't fifty be amazing? Usually not. With roughly 80% per-token acceptance, getting the first guess right is likely, but getting all five right in a row is much less so, and once an early token fails, everything after it is conditioned on a wrong prefix and gets thrown away. You still paid the draft model to produce it.

So raising K buys more potential progress per target invocation, more draft compute, larger verification blocks, more temporary KV pressure, and more wasted work after early rejections. There's an optimum, not a “bigger is better” curve. Longer speculation helps when acceptance is high, hurts when it's low, and the best setting shifts with workload and concurrency. K of 3, 4, or 5 are reasonable starting experiments, but there is no universal answer. You benchmark it.

“Use the most accurate draft model” is also wrong, or at least incomplete. Suppose your candidates are a 1B at 72% acceptance, a 3B at 80%, and an 8B at 88%, with relative costs of 1, 2, and 6 units. The 8B looks best on acceptance and may be worst on economics. You're not optimizing draft accuracy; you're optimizing something closer to accepted target tokens per unit of drafting-plus-verification time.

NVIDIA's own TensorRT-LLM numbers make the point. With Llama 3.1 70B as the target on an H200, they reported about 2.86× throughput with a Llama 3.2 1B drafter versus about 2.23× with an 8B drafter in that setup. The bigger, smarter drafter lost. The real rule isn't “a 70B should use a 7B.” It's that a 70B should use the cheapest predictor that's accurate enough to eliminate expensive 70B decode steps.

In a normal target decode step, the GPU streams enormous quantities of weights through memory to produce one token for one sequence. The useful arithmetic per byte moved is low. Implementations cache and optimize aggressively, but the utilization problem doesn't go away. Speculative decoding hands the target several positions to evaluate per weight load, so each expensive invocation does more useful work.

Which is why this technique is less about reducing theoretical FLOPs and more about cutting serial synchronization and improving effective hardware utilization. You often do more total arithmetic and still get lower wall-clock latency. That pattern shows up constantly in systems work: fewer operations doesn't mean faster, and more operations doesn't mean slower. Dependency structure and utilization decide.

If you come from systems programming, you already know this trick. CPUs don't wait for a branch condition to resolve before doing more work; they predict the likely path and start executing it. Correct prediction means the work is already done. Wrong prediction means you flush the speculative work and eat the cost.

Speculative decoding is the same philosophy applied to tokens. The draft says “I think the next tokens are brown fox jumps over.” The target checks all four. If they all pass, speculative work just became real progress. If walked fails at position three, everything after it is discarded and generation resumes from the corrected prefix. You deliberately do work that might get thrown away, because when it survives, it saves something more expensive.

If the target's distribution is sharply peaked, say token A at 0.95, any reasonably related draft will also propose A and acceptance stays high. If the distribution is flat, A at 0.20, B at 0.18, C at 0.17 and so on, the draft and target have far more room to diverge. Temperature, top-p, and top-k all reshape the distributions being sampled, so they directly change the acceptance pattern. Acceptance varies with decoding strategy and workload. Which is one reason you can't benchmark speculation on “Hello world” and assume the number transfers to production traffic.

Code is full of highly predictable local continuations: for i in range(len(items)):, if __name__ == "__main__":, repetitive JSON field structures, templated text. A small model approximates the large one well in these regions, so acceptance is high. A genuinely uncertain reasoning step (“Given these contradictory constraints, the optimal architecture would be...”) gives the target a next-token distribution that's much harder for the draft to match.

So acceptance isn't a property of the model pair alone. It's a function of the target, the draft, the prompt distribution, the generation phase, the sampling parameters, and the context length. A pair that flies on code completion can crawl on creative writing.

Running two models means storing two models, plus runtime state and KV cache for both. GPU memory has a price. If the draft eats memory that would otherwise go to a larger batch, more KV-cache capacity, or more concurrent requests, your single-request latency win can lower total server throughput.

That's why speculation looks most attractive for low-latency, relatively low-concurrency serving, and why the trade-off gets murkier once the server is already heavily batched. The overhead and the optimal speculation length both shift as concurrency rises. The general lesson: never optimize tokens/sec for one request and declare the serving system optimized. TTFT, TPOT, request latency, tokens/sec, requests/sec, GPU utilization, memory utilization, and cost per million tokens are different objectives, and speculation moves them in different directions.

If you want to actually understand this, don't hide behind vLLM or TensorRT-LLM on day one. Build a dumb version first. Take two Hugging Face models from the same family, one large as the target, one small as the draft, and write the loop yourself:

python
Copy
1while not finished:
2
3    # 1. Draft K tokens
4    draft_tokens = draft.generate_k(context, K)
5
6    # 2. Run target over the speculative block
7    target_probs = target.verify(context + draft_tokens)
8
9    # 3. Apply speculative acceptance
10    accepted = verify_and_sample(draft_probs, target_probs)
11
12    # 4. Commit accepted tokens
13    context.extend(accepted)

Don't optimize it. Instrument it. Log draft time, verification time, total generation time, drafted/accepted/rejected token counts, acceptance rate, mean accepted length, tokens per second, and time per output token. Then sweep K from 1 to 8 and temperature from 0 to 1.0, and swap workloads between code generation, summarization, factual QA, creative writing, structured JSON, and reasoning.

You'll start seeing the theory in your own numbers. Maybe K=2 is too conservative, K=4 is the sweet spot, and K=8 is wasted drafting. Then swap the draft model and watch a tiny drafter with 55% acceptance lose to a medium one at 79%, while a big drafter at 91% acceptance loses on cost. Somewhere in that sweep, speculative decoding stops being an ML concept and becomes a systems optimization problem, which is where it gets fun.

At minimum: acceptance rate vs K (it falls as K grows), tokens/sec vs K (which usually has a hump and an optimum somewhere in the middle), and speedup vs acceptance rate (roughly flat near 1× until acceptance gets high, then climbing fast).

The most educational single metric is average committed tokens per target verification. Baseline decoding gives you 1.0 by definition. If speculation gives you 3.7, you can see at a glance exactly where the latency reduction came from.

The failure modes are predictable once you have the model in your head:

Bad draft-target alignment. If the distributions diverge too often, acceptance collapses and you're paying for predictions that get discarded.

A drafter that's too expensive. A very smart drafter can produce beautiful predictions while destroying the economics. Draft accuracy is not system performance.

K too large. You speculate deep into the future and repeatedly throw away the tail.

High concurrency. If batching already saturates the GPU, there's less spare compute for verification to exploit, and coordination plus memory overhead start to bite.

Memory pressure. The second model can shrink KV-cache capacity or maximum concurrency.

Workload drift. You tuned on code completion, traffic shifts to creative writing, acceptance drops, and your optimization quietly becomes a regression.

That last one is why production systems should expose speculative metrics instead of flipping speculative_decoding=true and hoping. Accepted speculative token counts and mean acceptance length tell you whether speculation is still paying for itself.

The two-model design is just the easiest version to teach. The actual objective was never “run two LLMs.” It was: produce cheap, accurate guesses about future target tokens. Those guesses can come from anywhere. Modern systems use independent draft models, n-gram speculation, multi-token prediction heads, Medusa-style heads, and EAGLE/EAGLE-3. NVIDIA describes EAGLE-style approaches that speculate from features of the target's own hidden states, with no separate drafter required.

The durable abstraction is:

Copy
cheap proposer → candidate future tokens → expensive verifier → accepted target output

Once you see that, speculative decoding stops being one algorithm and becomes a family of inference architectures.

Why would adding a model make inference faster? Because normal decoding spends one expensive target invocation to make one step of progress, while speculation spends a little cheap compute to predict future work, and when the predictions land, one expensive invocation validates several steps at once. Instead of four serial 70B passes for four tokens, you get four cheap draft steps and roughly one 70B verification. Four expensive serial operations become one expensive operation plus some cheap ones. More models, less latency.

There's a general systems principle here. When an expensive operation is stuck being sequential, ask: can I cheaply predict future work and validate several pieces together? CPU branch prediction, database prefetching, cache read-ahead, network request speculation, out-of-order execution, and speculative decoding are all the same move. You willingly compute things that might be discarded. That looks wasteful if you count operations, but nobody ships systems optimized for minimum operation count. We optimize latency, throughput, utilization, and cost, and speculation trades some wasted cheap work for fewer expensive serial dependencies. That's the whole trick.

Skip the trivia version (“it uses a small model and a large model”) and keep these five ideas instead:

1. Autoregressive decoding is sequential: one target step normally yields one token, and that serial dependency is what's expensive.
2. During memory-bound, low-batch decode, the GPU has unused parallel compute sitting there.
3. Cheap speculation creates candidate future tokens, and once candidates exist, the expensive model can evaluate several positions in one pass.
4. Acceptance rate decides everything. High acceptance means cheap guesses replace expensive sequential work; low acceptance means cheap guesses become pure waste.
5. Proper speculative sampling is lossless with respect to the target distribution. The drafter proposes, the target decides, and the acceptance/correction math guarantees the output distribution stays the target's.

Picture a senior engineer reviewing a fast junior. Without speculation, the senior writes every line personally, which is expensive. With speculation, the junior writes lines 1 through 4 and the senior reviews all four at once. Often all four pass and you got four lines of progress for one review. Sometimes line 3 fails, the senior corrects it, line 4 becomes irrelevant, and work continues from the fix.

The junior isn't useful because the junior is always right. The junior is useful because the cost of guessing is much lower than the cost of the senior doing everything, and the guesses are right often enough. Swap junior for draft model, senior for target model, and review for parallel verification, and you have speculative decoding.

That's why one of the most counterintuitive optimizations in modern LLM serving is completely reasonable: to make your huge model do less serial work, give it a very fast model that guesses what comes next. The small model predicts, the large model decides, and the GPU finally has enough parallel work to make the expensive model move faster.

Evidence / further reading: Leviathan, Kalman & Matias, Fast Inference from Transformers via Speculative Decoding, demonstrated 2–3× acceleration on T5-XXL while preserving the model's output behavior. NVIDIA has reported workload-specific TensorRT-LLM results of roughly 2.2–2.9× for Llama 3.1 70B with different draft sizes, and up to 3.6× in its 405B tests, which shows both the upside and how much drafter selection matters. Controlled inference-handbook experiments similarly show that speedup is driven by acceptance rate, speculation length, model pairing, and hardware rather than any single universal configuration.