The first time I saw a serious LLM bill, I assumed it was a typo. It was not a typo. It was a feature working exactly as designed, and the design was "charge per token, and let the engineer learn what a token is the expensive way."
What's a token, really
A token is a chunk of text — roughly ¾ of a word in English. "Hello" is one token. "Tokenization" might be three. Punctuation and spaces count. You pay for tokens going in (your prompt) and tokens coming out (the response), usually at different rates, and output is often several times pricier than input.
// rough rule of thumb: 1 token ≈ 4 chars ≈ 0.75 words
// "The quick brown fox" -> ~5 tokens
// a 2,000-word doc -> ~2,700 tokens
// your 600-word "system prompt" you forgot about -> ~800 tokens, every callNote
Cost ≈ (input tokens × input price) + (output tokens × output price). Latency tracks tokens too — more tokens, slower response. Tokens are the unit of literally everything that matters.
Do the napkin math first
Before optimizing anything, estimate. Say each request sends a 1,000-token prompt and gets a 500-token answer, and you serve 100,000 requests a month. That's 100M input + 50M output tokens. Plug in your provider's rates and you'll often find one bloated field — usually the system prompt or dumped context — is responsible for most of the bill. Optimization without estimation is just superstition.
Where the tokens actually go
When the bill spikes, it's almost never one giant request. It's death by a thousand prompts, each quietly carrying baggage:
- A system prompt that grew to 700 words of instructions, examples, and one passive-aggressive note from a former teammate — sent on *every* call.
- Whole documents pasted into context when you needed two paragraphs.
- Full chat history replayed on every turn, so message #30 is dragging 29 ancestors behind it.
- Verbose outputs because you never told the model to be brief, so it writes you an essay with a conclusion.
How to actually cut it down
1. Put the prompt on a diet
Models in 2026 are good. They don't need you to over-explain. Cut the throat-clearing, keep the constraints.
// before: ~120 tokens of vibes
const system = `You are a helpful, friendly, professional assistant. Please be
kind and thorough and make sure to always do your absolute best to help the
user with whatever they need, taking care to be accurate and detailed...`;
// after: ~25 tokens of instruction
const system = `Answer concisely and accurately. If unsure, say so. Cite sources.`;2. Retrieve, don't dump
This is RAG's other superpower: instead of pasting a 30-page handbook into context, retrieve the 3 relevant chunks. Same answer, a fraction of the input tokens, and often *better* because the model isn't drowning in irrelevant text. You can cut input tokens by 90% and improve quality at the same time — rare to get both.
3. Summarize the conversation
Don't replay the entire chat forever. Keep the last few turns verbatim and compress the older stuff into a running summary. Your message #30 stops paying rent on messages #1 through #25.
function buildContext(history: Msg[]) {
const recent = history.slice(-6); // keep recent turns sharp
const older = history.slice(0, -6);
const summary = older.length ? summarize(older) : ''; // compress the rest
return summary ? [{ role: 'system', content: `Earlier: ${summary}` }, ...recent] : recent;
}4. Cache the repeats
If the same question or the same big system prompt shows up constantly, cache it. Prompt caching (where the provider lets you reuse a fixed prefix at a discount) plus a plain response cache for common queries both quietly delete tokens you were about to pay for again. Identical FAQ questions shouldn't hit the model twice.
5. Right-size the model
Not every task needs your most powerful (and most expensive) model. Classification, extraction, and routing often run perfectly on a smaller, cheaper, faster model. A common pattern: a cheap model triages and handles the easy 80%, escalating only the hard 20% to the expensive one.
// route by difficulty: cheap model first, escalate only when needed
const draft = await cheap.complete(prompt);
if (draft.confidence < 0.7) {
return expensive.complete(prompt); // worth the tokens here
}
return draft;Tip
Cap your output. max_tokens plus a clear "answer in 2 sentences" is the laziest, most effective cost cut there is. The model will happily ramble if you let it.
Measure it like an engineer
Log tokens per request. Track input vs output. Find your top 5 most expensive endpoints and fix those first — the same 80/20 that governs every other performance problem you've ever had. You can't optimize a number you refuse to look at.
logger.info('llm', {
endpoint,
model,
inTokens: usage.prompt_tokens,
outTokens: usage.completion_tokens,
costUsd: estimateCost(model, usage),
});
// then: group by endpoint, sort by total cost, fix the top of the listTokens are just bytes with a price tag. Treat them like you'd treat any other resource you're billed for — count them, trim them, cache them — and your LLM bill goes from "please explain this to finance" to "rounding error."
Key takeaways
- 01Tokens are the unit of cost, latency, and context — count them or pay the price.
- 02Bills usually balloon from bloated prompts, dumped documents, and replayed history.
- 03Trim prompts, retrieve instead of dump, summarize history, cache, and right-size the model.
FAQ
Related reading