Cost Optimization for AI APIs

LLM API costs scale with tokens, and tokens scale with usage. A prototype that costs pennies per user becomes expensive at ten thousand users. Most teams can cut their bill substantially without changing what the product does. This guide covers the techniques in order of typical impact: prompt caching, model routing, batch APIs, semantic caching, token discipline, and output length control. Pricing mentioned here is the discount structure published by the providers, not per-model rates, which change.

Prompt caching

If your requests share a long common prefix, prompt caching is usually the single biggest lever. Anthropic's prompt caching is explicit: you mark a breakpoint with cache_control: {type: "ephemeral"} and the provider reuses the computed state for that prefix on later calls. Cached reads cost 10% of the base input price, a 90% discount, while the first write costs 1.25 times the base input price. Break-even arrives fast when the prefix is large and repeated. The cached prefix must be byte-identical and at least 1,024 tokens for the models this applies to, and the cache lasts about five minutes, refreshed on each hit.

```python

response = client.messages.create(

model="claude-sonnet-4-5",

system=[

{

"type": "text",

"text": LONG_SYSTEM_PROMPT,

"cache_control": {"type": "ephemeral"},

}

],

messages=[{"role": "user", "content": user_input}],

)

```

OpenAI applies prefix caching automatically with no configuration: repeated prefixes of 1,024 tokens or more get a 50% discount on those input tokens. No write surcharge, no markup needed. Check the cached_tokens field in the response usage to confirm it is working.

Two rules make or break caching on either provider. Put all static content first and dynamic content last, because caching works on prefixes. Keep the prefix byte-identical across calls. A timestamp, a random request ID, or a user's name placed before the cached block invalidates the whole cache. Monitor cache creation versus cache read tokens in your usage logs; heavy creation with no reads means your prefixes are unstable.

Model routing

Not every task needs the strongest model. Route simple tasks to small cheap models and escalate only when needed. The two patterns that work: classification first, and try cheap then escalate.

In classification first, a small model labels the incoming request by difficulty or type, then the router sends it to the appropriate model. In try cheap then escalate, the request goes to the small model first, and a verifier (a cheap check, a judge call, or a confidence threshold) decides whether the answer is good enough. If not, the request goes to the larger model.

Build the router on your own traffic. Label a few hundred real requests by which model handled them well, and you have a training set for the classifier. Measure the escalation rate and the cost per request against the all-flagship baseline. A router that escalates 30% of traffic to the flagship and handles 70% on a model that costs a tenth as much cuts the input bill roughly in proportion, as long as answer quality holds on your eval set.

Batch APIs

Work that does not need an answer now should not pay real-time prices. OpenAI's Batch API and Anthropic's Message Batches both price asynchronous requests at 50% off standard token prices, on input and output. Batches complete within 24 hours, usually much faster, and use a separate capacity pool with much higher rate limits.

Good candidates: evaluation runs, offline classification, summarization pipelines, synthetic data generation, embeddings backfills, anything with a cron schedule. Both providers take a JSONL file of requests with a custom ID per line and return per-request results you can poll for. If your monthly API spend is in the thousands and a third of it is offline work, the batch discount is the easiest money on this list.

Semantic caching

When many users ask the same questions in different words, cache by meaning instead of by exact text. Store embeddings of incoming queries alongside their responses. For a new query, search for a close match above a similarity threshold. On a hit, return the stored answer with no model call at all.

This works for support bots, documentation Q&A, and anything with a long tail of repeated questions. The threshold is the whole game: too low and you serve wrong answers, too high and the hit rate collapses. Tune it on labeled pairs from your own logs, and exclude anything time sensitive or user specific. A cached answer about last quarter's pricing policy is a liability. Set a TTL and an invalidation path for content that changes.

Token discipline

Every token you send costs money, including the ones you never read. Count tokens before calling the API, not after the invoice arrives. Both providers' SDKs expose token counting: OpenAI's tiktoken library and Anthropic's count_tokens, or the usage field on every response.

Trim ruthlessly. Remove few-shot examples that do not change quality; test with two, then one, then zero. Replace a 5,000-token reference document dumped into every prompt with retrieval that sends only the relevant chunks. Strip chat history to the turns that matter instead of the whole conversation; summarize older turns when the task needs context. Put a per-request input token budget in code and alert when requests exceed it. A single pathological input, like a user pasting a novel, should be truncated or rejected, not billed.

Output length control

Output tokens usually cost several times more than input tokens on the same model, so the answer length is where the money goes. Set max_tokens explicitly on every call. Instruct the model to be concise in the system prompt for tasks where brevity is acceptable. For structured outputs, ask for only the fields you need instead of a narrative plus the fields. A classification call should return a label, not a paragraph.

Watch for runaway generation in production. Log output token counts per request and alert on outliers. A prompt change that makes answers twice as long doubles the most expensive part of the bill silently.

The order of operations

Start by measuring: per-request token counts, cost per request, and which endpoints spend the most. Then apply the levers in this order. Turn on prompt caching for any endpoint with a shared system prompt, it is configuration, not engineering. Move offline work to batch APIs for an instant 50% cut on that slice. Build the model router for your highest volume endpoint. Then do the harder work of token discipline and semantic caching. Re-measure monthly. Costs drift as usage patterns change, and a new endpoint can quietly become your biggest line item while nobody is watching.