Skip to main content
Engineering notes 9 min read

The Optimisation I Had Already Rejected, Twice

Forty-two percent of my answer time went on the model retyping text out of its own prompt, so I wired up the technique built for exactly that. It came out 44% slower. The payoff rides on one ratio — about 15% in my output — and I had already measured it, days earlier, into a flag I never read.

A grounded answer with numbered citations and source cards quoting the recordings
The answer format this optimisation was aimed at: prose over quotes lifted from the prompt.

I measured where the time goes when my app answers a question about someone’s recordings. Retrieval — finding the passages worth showing the model — is 18%. Prefill, the model reading that prompt before it writes a word, is 30%. Writing the prose answer is 9%.

And 42%, the single largest cost in the pipeline, is the model retyping quotes out of passages it was just handed.

Forty-two percent of the wait, spent copying text that is already sitting in front of the model. There is a technique built for exactly that. I called it the headline finding of the session, wrote the code, wired it into both chat paths, typechecked it, and launched an evaluation run.

Then I opened an unrelated file and found the technique already in the codebase, behind a flag, off by default. Its documentation is one line, and the line is a verdict: measured twice and rejected — 11.4 minutes of compute per hour of recorded audio against 6.5 minutes with it off, and 8% of guesses accepted on the chat path.

I hit this building the library chat in Kajo Voice, where every answer is generated on the user’s own machine, so decode time is wall-clock time a person spends waiting.

The measurement behind the second half of that line is the part worth reading, because the analysis that produced it was good. I checked the three things that usually kill this technique, all three came back clean, I wrote the code, and I measured: 44% slower. Everything in the analysis was right except one number, and that number is the mechanism. I never estimated it.

The bargain speculative decoding makes

A language model writes one token at a time — a token being roughly a short word or a piece of one. Each token costs a full pass through the model’s weights.

What makes that pass expensive is mostly hauling those weights out of memory and into the processor, not the arithmetic performed with them once they arrive. So a pass that evaluates eight positions at once costs barely more than a pass that evaluates one. Generating strictly one token at a time leaves most of that capacity idle.

Speculative decoding buys it back. Something cheap guesses the next few tokens. The real model then evaluates all of them in a single pass and checks, position by position, whether it would have produced each one itself. It keeps the longest run of guesses that agree with it and throws the rest away.

Two things follow from that, and I acted on the first while skipping the second.

The output stays honest. Every token that survives verification is a token the model would have produced anyway, so the technique cannot make an answer worse or a citation wrong. That is why it is safe to switch on, and why it gets described as free speed.

But the speed is conditional, and the condition is a ratio. A pass that verifies eight guesses and keeps six has produced six tokens for a little more than the price of one — a large win. A pass that keeps none has produced exactly one token, the one it was always going to produce, for a little more than the price of one. Rejected guesses are not free: the compute spent checking them was spent.

So the payoff scales with the share of guessed tokens that survive verification. Not with whether guessing sounds plausible for your workload. With the share that survives.

Prompt-lookup decoding is the variant I wanted. It replaces the cheap guesser with plain string matching — no second model involved at all. When the text the model has just written also occurs somewhere in the prompt, the matcher copies whatever followed it there and offers that as the guess. It costs nearly nothing, and it is precisely right when the model is retyping its input. node-llama-cpp, the runtime I ship, exposes it as a token predictor, and its documentation names input-grounded tasks — where the model frequently repeats input tokens in its output — as the ideal case.

Why it looked perfect

My answers are input-grounded by any description you like. Each one is a JSON object carrying an array of verbatim spans copied out of the retrieved passages, plus prose that cites them. The 42% is not an incidental cost; it is the product’s main output being typed back out of the prompt, one token at a time, while somebody waits.

Before writing code I checked the three things that usually kill this.

Can the runtime verify a batch at all? The multi-token-prediction path is compiled into the llama build I ship, so it can evaluate several guessed positions in one pass. Without that there is no mechanism for a speedup at all.

Can a guess break my output format? My output is grammar-constrained: the runtime is handed a description of the exact JSON shape I want and will not emit a token that violates it. I expected speculation to fight with that. It does not — speculative evaluation clones the grammar state and tests every guessed token against it, so a guess that would break the shape is simply rejected like any other wrong guess.

Has it been tried here? Once, elsewhere in the codebase, and switched off again after measuring 11.4 minutes of compute per hour of recorded audio against 6.5 minutes with it off. The stated cause looked specific to that path: speculation competes with multi-sequence batching for the same batch capacity, and that path runs three generations at once. The chat path runs one. No contention to speak of.

A specific, mechanistic, testable hypothesis, with the one counter-example on the record explained away. A three-minute probe killed it.

median answer time output rate guessed tokens
predictor off 3,688 ms 250 characters/second —
predictor on 5,302 ms 170 characters/second 19 accepted / 226 rejected — 8% accepted

Nineteen guesses survived verification and 226 were thrown away, each of them having cost its slice of a real pass.

The contention story was correct and also a red herring: there was no contention on this path, exactly as I had argued, and it made no difference at all. At an 8% acceptance rate, nearly every pass produced the single token it was going to produce anyway and paid extra to do it.

The number I never estimated

Here is the arithmetic that takes a minute.

The quotes are 6 to 12 words each, and those are copied. Everything wrapped around them is JSON scaffolding — {"evidence":[{"snippet_idx":1,"quote":" and its closing punctuation — which appears nowhere in the prompt. And the answer field is freshly written prose, which by definition appears nowhere in the prompt. Only the spans inside the quotes can ever be guessed correctly.

Call it 15% of output tokens. That is the share the mechanism gets paid on, and an 8% acceptance rate was never going to support speculation.

I had the real figure available without doing any arithmetic at all. The library reports the ratio directly, as validated and refuted counters on every generation. The estimate I skipped was a field read.

What I did instead was reason about the shape of my output — “it copies quotes out of the prompt, so it must be mostly copied” — and never convert that shape into a fraction. The shape argument is true. It is also compatible with 15%.

Every claim of the form “my workload is the ideal case for this” is a claim about a ratio, and the ratio usually goes unnamed. “Input-grounded” is a claim about the share of output tokens already present in the input. “Cache-friendly” is a claim about hit rate. “Embarrassingly parallel” is a claim about the fraction that is genuinely independent. Name the ratio the mechanism gets paid on, then go and read it — the runtime is usually counting it already, under a name you have read straight past.

One correction to something I asserted while setting this up, because it falls straight out of the mechanism above. The library says a token predictor “doesn’t affect the generation output itself”, and I repeated that as “output is identical”. Under a fixed seed the outputs differed — 892 characters against 919. Speculative decoding is distribution-preserving, not byte-identical: accepting a batch of verified guesses consumes the random number stream differently than sampling the same tokens one at a time. Neither output was wrong; the word “identical” was, and a byte-for-byte regression test resting on it would have failed for a reason unrelated to anything it was written to catch.

The record I never searched

Two days separated that measurement from the session at the top of this post, where I produced the latency breakdown, found the predictor in the vendor’s documentation, and rebuilt it: the same reasoning, the same three mechanism checks, the same conclusion, the same correction about byte-identical output, all re-derived from scratch. Every part of it was already written down three times — in a lessons file, in the environment reference, and in a comment at the call site — and my write-up of the first attempt sat in the same repository.

The idea was never the failure. The idea had been evaluated properly, twice. The failure is that I spent the whole research budget of that session on the vendor’s documentation and the open web, and never searched my own repository for the name of the thing I was about to build. That search costs ten seconds. Vendor documentation describes a technique’s ideal case; a lessons file, an environment reference and a constant’s comment describe what it did here. A technique already sitting behind an off-by-default flag has been measured and rejected, and the flag’s documentation usually says why.

Prompt-lookup decoding is a good technique. Its authors report a relatively consistent 2.4x, on average, on summarisation and context-heavy question answering, and turn-dependent gains on multi-turn chat — workloads where the output really is mostly copied input. It did not fit here, and the reason was legible in advance: in a counter the runtime had been printing the whole time, and in one line of my own documentation.

Ready to keep your archive local? Pay once — $49 through December 9, 2026, and $99 after that. No subscription. Or start free.

Compare Free and Lifetime →

Related articles