Estimate LLM Token Costs Before You Ship the Feature
Token bills sneak up after the demo looks cheap. Count tokens, model price, and retries before you put an LLM in a user path.
Part of the AI for Business Operations series.
The prototype answers three questions a day. Production answers three thousand - plus retries, plus a retrieval chunk you forgot to meter, plus a system prompt that grew every sprint. Finance notices before engineering does.
Estimating tokens isn’t glamorous. It’s how you avoid shipping a feature whose unit economics only work in a slide deck.
What actually drives the bill
- Input tokens - system prompt, tools schemas, conversation history, RAG chunks.
- Output tokens - especially if you let the model ramble without max tokens.
- Multiplier paths - retries, parallel tool calls, evaluation loops, “agent” steps.
- Model tier - the smart model for every keystroke is how budgets die.
Estimate in an afternoon
- Collect 20-50 realistic prompts (include ugly ones).
- Measure them with Token Counter or Context Estimator.
- Plug volumes into LLM Cost Estimator and sanity-check alternatives in Cost Compare.
- Multiply by expected daily users, then by a retry factor (1.2-2.0 is common while you stabilize).
- Add embedding costs if you retrieve; they’re small per call and large in aggregate.
Design knobs that cut spend
- Cache stable system instructions server-side when the vendor supports it.
- Trim history; summarize older turns instead of sending the novel.
- Cap output tokens for classification and extraction jobs.
- Route easy tasks to a smaller model; reserve the expensive one for hard cases.
- Scan prompts for accidental PII before they leave - Prompt PII Scanner - so you’re not paying to process secrets you shouldn’t send anyway. Pair with habits from using generative AI safely.
A ship gate for product + eng
No LLM feature merges to prod without: estimated monthly cost at target volume, a kill switch, logging of token usage per request type, and an owner who watches the first two weeks of invoices.
Scenario planning beats single-point estimates
Build three columns: conservative, expected, stress. Stress should assume viral usage or a retry storm. If stress economics only work when nobody uses the feature, you don’t have a feature - you have a demo.
Include support burden loosely: if the model is wrong 10% of the time and a human must intervene, that’s cost too. Token invoices aren’t the only line item.
Instrumentation to add on day one
- Tokens in / tokens out / model id per request.
- Feature flag or route name for attribution.
- Cache hit rate if you cache prompts or embeddings.
- Daily budget alarms before finance forwards the bill.
Without instrumentation you’ll argue from anecdotes. With it you can tune routing to cheaper models using evidence.
Product constraints that save money
UI copy can limit abuse: rate limits per user, clear “generate” buttons instead of firing on every keystroke, and defaults that don’t attach entire knowledge bases to each call. Cost control is partly a UX problem.
Field notes from teams who shipped this
The pattern that keeps showing up: write the constraint first, then the steps, then the failure modes. Teams that only publish happy-path screenshots create tickets. Teams that document the ugly path create trust.
Schedule a short review ninety days after publishing. Check whether product UI names still match, whether linked tools still exist, and whether support still hears the same questions. Update the page or merge it. Standing still is how useful posts become interchangeable again.
If you adapt this article for internal wikis, keep the examples tied to your stack names. The moment you generalize back to “best practices for organizations,” you’ve started erasing the specificity that made the piece worth saving.
RAG-specific cost notes
Retrieval adds embedding costs and larger prompts. Measure average chunks per query and the token size of each chunk. A “helpful” top-k of 12 can dominate spend if each chunk is fat.
Try smaller chunk sizes, better filters, and hard caps on context characters. Log when the model answers without using retrieved context - you may be paying for decoration.
Evaluate whether FAQ-style deterministic answers should bypass the LLM entirely. The cheapest token is the one you never send. Product judgment beats model enthusiasm here.
Revisit estimates after the first month of real traffic; prototypes rarely predict which prompts users invent under deadline pressure.
Contract caps and overage
Know your vendor’s overage pricing before launch marketing spikes traffic. Soft caps with alerts beat hard outages, but finance still wants a number. Put the alert in the same channel as deploy notifications so eng sees cost beside reliability.
Re-estimate when you change models - price boards move, and last quarter’s spreadsheet lies quietly.
Stakeholder one-pager
Translate the estimate into a single page for non-engineers: expected monthly cost at target usage, stress-case cost, kill switch, and the date you’ll revisit the model choice. Executives don’t need tokenizer details; they need bounded risk.
Include one sentence on quality tradeoffs if you pick a cheaper model. Surprises about “worse answers” hurt trust more than a known cost line item.
Attach the spreadsheet or estimator screenshot so the number isn’t tribal. When pricing pages change, update the one-pager in the same week.
FAQ
Our counts don’t match the vendor invoice.
Different tokenizers and billing rules exist. Use vendor counters for final numbers; local counters are for directionally honest planning.
How do I estimate agents?
Meter each step. Worst case = max steps × tokens per step × price. Optimistic demos lie; budgets shouldn’t.