The cheapest token is the one you never send

Automatic speech recognition gets you eighty percent of a transcript. The last twenty percent is where the meaning lives: sentence boundaries, punctuation, and above all the proper nouns — a Sanskrit name that Whisper heard as three unrelated Russian words. So every raw transcript goes through a proofreading pass with a language model before it reaches a reader.

At our measured rate that pass cost $0.106 per lecture. With 3,900 lectures left in the archive, the bill came to roughly $400. Not ruinous, but large enough to ask whether we were spending it well.

We were not. Here is what the money was actually buying, and what happened when we looked.

First, the obvious idea, and why it failed

The reflex is to find a cheaper model. Open weights are a fraction of the price per token, and the task looks mechanical — fix spelling, do not paraphrase.

We had an unusual advantage for testing this. Around a third of the archive carries a transcript written by a human editor, published alongside the audio. That gave us an answer key: run a candidate model over the raw ASR of those lectures and measure how much closer to the editor’s text it lands. Not a judgement call — a number.

Five models, three lectures each, same prompt, same chunking:

The cheap models are cheap because they barely do the work. gpt-oss-20b failed the output contract on 54 of 56 chunks — it would not reliably return the segment set it was given. gpt-oss-120b held the contract perfectly and then changed almost nothing, moving the error rate by a third of what the incumbent managed. And because it is a reasoning model billing its thinking at the output rate, it came out more expensive than the model it was supposed to replace.

Cost per token is the wrong denominator. Cost per error actually fixed is the right one, and by that measure the cheap models were the expensive ones.

Where the money was really going

So we stopped looking at models and looked at the invoice. On our baseline model input costs $0.25 per million tokens and output costs $1.50 — output is six times dearer. Measured across three lectures:

input    85,493 tokens   $0.0214   23%
output   46,710 tokens   $0.0701   77%

Three quarters of the bill was the model writing. So what was it writing?

The contract at the time asked for strict JSON: every segment of the chunk returned with corrected text, plus an array grouping those segments into sentences. We decomposed a real reply:

Only the second row is work. We were paying premium output rates for the model to dictate back to us, verbatim, the half of the transcript it had decided was already correct — and to wrap every line in JSON punctuation while doing it.

Ask for the diff, not the document

The fix is the one every version control system made obvious decades ago: send back what changed.

The reply format became plain lines. One line per segment the model actually corrected, then the sentence boundaries:

11|Шри Прабхупада говорил об этом.
13|И потому мы читаем «Бхагавад-гиту».
ENDS
11,14

Omission means “leave this segment alone”. A chunk needing no changes returns nothing but the boundary line.

The ENDS line deserves its own note, because it is the same lesson twice.

We ask for sentence ends at all because the stored transcript is a list of sentences with timings, and Russian punctuation alone will not give you them: abbreviations and initials carry periods that end nothing, a quoted Sanskrit verse belongs inside the sentence that introduces it, an ellipsis may be a pause. The model has to judge it.

What changed is how much of that judgement we pay to hear. The old contract asked for full sentence membership — [[10,11],[12,13,14]] — and the code consuming it kept only the last index of each group, discarding the rest. Sentences are runs of consecutive segments, so those boundaries alone rebuild the grouping exactly. We had been buying data we deleted on arrival; now the model sends 11,14 and nothing more.

Measured on the same three lectures, each variant against the human answer key:

Accuracy did not move — and structural failures went to zero. A model asked to re-dictate 150 segments in strict JSON loses the thread sometimes; a model asked for eight short lines does not.

In production, across 100 lectures and 4,415 chunks, the cost per chunk fell from $0.00527 to $0.00123 — a 77% cut. Escalation to the expensive fallback model dropped too, from 34% of chunks to 23%, because fewer replies broke the contract.

The thing that did not work, and why it is interesting

With output tamed, input became the dominant cost — 60% of the bill. We fitted a line through 1,043 production chunks:

input tokens ≈ 1,482 + 0.465 × characters of transcript

Two thirds of the input was a constant re-sent on every single call: the system prompt, the user template, the tail of the previous chunk, the glossary hints. That is exactly what prompt caching exists for.

It refused. Gemini’s context cache has a minimum of 1,024 tokens and our system prompt is 558. We had spent the previous hour trimming that prompt by 40% for savings — and in doing so had put it firmly out of reach of the cache. Two optimisations pulling in opposite directions, and the honest answer was to take the smaller certain win and drop the caching idea rather than pad the prompt back up to qualify for a discount on words we did not need.

Half price for patience

The last lever costs nothing but time. Batch endpoints run your requests on spare capacity within a 24-hour window and bill at half the synchronous rate. A corpus import has no latency requirement whatsoever.

The interesting part is what it did to throughput. Job size barely matters:

Queue depth dominates, not volume. Splitting the corpus into small batches would be strictly worse — each job pays its own wait.

Three things about batch APIs are worth knowing before you trust one, all of which we learned by being bitten:

• A job reports success with every request inside it failed. The job state answers “did it stop moving”, not “did it work”. Read the per-request stats.
• A dropped request is simply absent from the replies. Nothing errors. You find it by subtracting the keys you sent from the keys you got back.
• Our model rejects thinkingBudget: 0 outright — and spends no thinking tokens without it. We burned an entire job of 27 requests on an optimisation that was never needed.

That last point cost us six minutes and taught the general lesson: verify the parameter does something before you set it everywhere.

Because the replies are addressed by a key we choose (track:chunk), the stragglers are trivial to recover. Whatever the batch failed to deliver gets proofread live during collection, in the same pass — a few cents rather than another 24-hour wait. Across 3,214 chunks run this way, 22 needed the live path and none were lost.

What it adds up to

per lecture                                        per chunk

$0.1086   full JSON, synchronous            1×     $0.00527
$0.0544   line deltas, synchronous        2.0×     $0.00123
$0.0146   line deltas, batch, chunks of 50 7.4×    $0.00075

Ten point nine cents a lecture became one and a half. Seven times less.

Across the 3,900 lectures still to process that is $423 down to $57 — and the run behind these numbers, 1,543 lectures in a single evening, came in at about $22.

Not one cent of that came from a cheaper model. It came from noticing that we were paying, at the most expensive rate on the invoice, for a model to repeat text we had sent it, wrap it in punctuation we discarded, and enumerate groupings we reduced to their last element.

The pattern generalises past our particular pipeline. When a model call feels expensive, the useful question is not which model is cheaper but what fraction of the response is information we did not already have — and whether the format you chose is quietly billing you for the rest.