Someone asked me a fair question about the search in my app: if we improve the way transcripts get cut into pieces, does the keyword half of search get better too?
I answered from the formula. Twenty minutes later it was a task in the plan with a fix attached, and I had told them it was a real finding. Then I measured it, and it was wrong in three separate ways — the third of which made the task pointless.
The argument I made is the textbook one. That is what makes it worth walking through slowly.
I hit this building retrieval for Kajo Voice, a local-first audio knowledge base that runs a keyword index and an embedding model side by side on your own machine, over your own recordings.
The argument, as I made it
BM25 is the scoring function behind most keyword search, including SQLite’s built-in full-text index, which is what I ship. For a given search term it pushes a document’s score up for two reasons and down for one.
Up: the term appears many times in this document. Up: the term is rare across the whole collection, so matching it is informative. Those two are the familiar half.
Down: this document is long. The intuition is good — a word appearing once in a tweet tells you more about the tweet than the same word appearing once in a novel tells you about the novel. So BM25 divides by the document’s length relative to the collection average, and how hard it divides is set by a parameter called b.
Now the argument I built on it. In my library, a passage of Chinese or Hindi transcript measures about 1.9 times the length of an English one, because of how the text gets split into tokens before indexing. Long documents are penalised. Therefore, in a library holding both, non-Latin recordings are quietly ranked below English ones, and the fix is to turn b down so the penalty bites less.
That reads like a finding. Here is what measuring did to it.
The knob I proposed does not exist
In SQLite’s full-text engine you get BM25 by calling bm25() inside a query. It takes the table, and then some more arguments. I assumed those were k1 and b, the two standard BM25 parameters.
They are not. The extra arguments are per-column weights — they let you say “a match in the title counts double”. k1 and b are compiled into the engine, with no way to set them from a query. So bm25(t, 1.2, 0.0) and bm25(t, 1.2, 0.3) return byte-identical scores: the value I passed was read as a column weight and ignored.
A b of zero switches length normalisation off entirely. Nothing moving should have been my first clue, and one query would have found it. A knob you have not called is a knob you have not got, and a plan resting on an imaginary parameter is worse than no plan, because someone will eventually try to execute it.
So I moved the collection instead
With b out of reach I needed another way to turn the length penalty up and down, and the formula supplies one: BM25 divides by a document’s length relative to the collection average, so moving that average changes the penalty on every document without touching any document.
I set up four candidate documents, each containing the search term exactly once, plus 80 filler documents so the rarity weighting behaves like a real index. Then I changed nothing about the candidates — I only swapped the filler, so the collection average length moved from about 295 tokens to about 161.
The four candidates are h1 to h4, listed best match first. SQLite returns BM25 as a negative number, so more negative means a better match.
| collection average | resulting order |
|---|---|
| ~295 tokens | h4 (−14.359) h1 (−9.524) h2 (−5.120) h3 (−3.425) |
| ~161 tokens | h4 (−10.937) h1 (−7.093) h2 (−3.612) h3 (−2.740) |
Every score shrank. Every score shrank by roughly the same factor. The order is identical.
While I was in there, my headline number went too. I had said a document at twice the average length loses roughly 40% of its score. Measured, with the same single occurrence of the same term, it keeps 74.6% — it loses about 25%, and the closed form agrees to four decimal places, so this was not a subtlety about my engine. It was mental arithmetic in support of a conclusion I had already committed to. A quarter is a real penalty, but it is the sort of gap a strong match closes easily; 40% sounded like something to act on this week.
Rank is the only thing my pipeline reads
My retriever searches two ways for every question — the keyword index above, and an embedding model that matches on meaning rather than words. Each way returns its own ranked list, and something has to merge them.
The standard way to do that is Reciprocal Rank Fusion. It is deliberately crude: each list contributes 1 / (k + position) for each document, and the merged list sorts by the sum. A document that came first in the keyword list contributes the same amount whether it won by a hair or by a mile. The scores are discarded at the door; only 1st, 2nd, 3rd survive.
So multiply every keyword score by 0.76 and nothing downstream can tell. The ranks are unchanged, so the fusion inputs are unchanged, so the merged list is unchanged, so the passages handed to the model are unchanged.
That is the whole defect, dissolved. The length penalty is real, it is measurable, and in my system it lands on a quantity that gets thrown away one stage later.
The reason the factor is shared across candidates — rather than hitting some and not others — is that a Chinese query only matches Chinese text, because of how both get tokenised. So nearly every query is a contest between documents of similar length, in one script, all taking the same penalty together.
What survived
Not nothing, but something much smaller.
Cross-script matching does happen, on tokens the scripts share: proper nouns, numbers, product names, code identifiers. Search openai and a Chinese passage and an English one can genuinely compete — and there the longer one really does lose. One occurrence in a 295-token passage scores −3.450 against −2.515 in a 154-token one. It keeps 72.9% of the score, and this time it loses the rank too.
That is one narrow class of query, on one half of a retriever whose other half is a multilingual embedding model that scores a script-neutral token about the same either way, and fusion already covers most of it. So it went back on the list as “re-open if a measurement ever shows shared-token queries failing” — and if that day comes the fix belongs on the query side, never in the chunker. Reshaping passages to please keyword search would degrade the embedding model to help its partner.
The same mistake, inverted
Asking what reads a number next also catches the opposite error, and I had one of those in the same pipeline.
After fusion, a cross-encoder reranker reads the question and each candidate passage together and reorders them properly. Mine had a floor: if the best score it produced for any candidate came in under 0.15, the pipeline discarded the reranker’s ordering and kept the fusion order, on the theory that a reranker scoring everything that low “has no opinion”.
The model I use scores a perfect match at about 0.08.
I checked 24 test questions against the exact passage each one was written to find — the most favourable pairing that exists — and 21 of the 24 came in below the floor, median 0.000199. The pipeline had been overruling its own reranker on almost every question for as long as that constant had existed. The comment above it contained its own refutation: it explains that you cannot threshold individual candidates because the scores are arbitrary from query to query, then thresholds the maximum of those same scores.
Removing it, 70 of 80 test questions got their intended passage in front of the model, against 69 with it in place. A small gain — but all three questions it recovered were about what one named person said, which is the case the reranker exists for.
Two mistakes, one shape. In the BM25 case the absolute value did not matter and I built a task on it. In the reranker case it was the only thing that mattered and I had borrowed the number from nowhere — a constant compared against a model’s output belongs to that model, not to the pipeline, so swapping the model makes it stale. Which is how it surfaced: a model comparison that was really comparing “one model, mostly ignored” against “another model, actually used”.
Both came from reading a number without asking what consumes it. That question needs no benchmark, and asking it would have cost me less than either write-up did.