LLM Cost Optimization: The Complete Guide (2026)

The invoice landed on the first of the month, and it was three times what the team had modeled.

They'd done the homework. Someone had pulled every provider's pricing page into a spreadsheet, sorted by dollars per million tokens, and picked the cheapest model that cleared the quality bar. On paper the math was airtight.

Then production happened. The retrieval got richer. The prompts got longer. The agents started calling tools that fed more context into the next call. And the one number they had optimized for, the sticker price, turned out to be the one number that barely mattered.

If you've ever stared at an AI bill that didn't match your estimate, this guide is for you. LLM cost isn't random. It comes down to four levers, and all four are knowable and tunable. By the end you'll be able to model your own bill instead of guessing at it.

The four levers of LLM cost. Workload shape sets your input-to-output token ratio, and output runs 3 to 5 times the price of input. Model choice only makes sense once you know that ratio, because the cheapest of 1,000+ models flips between input-heavy and output-heavy work. Prompt caching takes 50% to 90% off a stable prompt prefix, but only at the hit rate your traffic supports. Batch processing takes roughly 50% off anything that can wait up to 24 hours. And traffic multiplies all four: a $12,000/month workload costs $120,000 at a 10× spike.

The number you're optimizing barely matters

Price-per-token is real, it's published, and it's the easiest thing to compare, which is exactly why everyone anchors on it. Comparing it feels like diligence.

But price-per-token is an input, not an outcome. Your actual bill looks like this:

(input tokens × input price) + (output tokens × output price), multiplied by every request you serve.

Every term in that equation except the price is something you control, and most of them dwarf the price gap between two comparable models. Two teams running the "same" model on the "same" use case routinely land on bills that differ by 5 to 10 times, because their token math is completely different.

So stop leading with the price tag and start with the four things that set your token math.

Lever 1: Workload shape, the biggest driver of them all

Every request has two halves, and they're priced separately.

Input is everything you send the model: the system prompt, the instructions, the retrieved documents, the chat history, the user's question. It's usually the larger, cheaper half.

Output is what the model generates back: the answer, the code, the summary. It's almost always the pricier half.

How much pricier? On most frontier models, output costs 3 to 5 times more per token than input. GPT-4o, for example, charges $2.50 per million input tokens and $10.00 per million output. That's a 4x gap on the same request.

That gap shows up fast. Take a simple chatbot turn: 800 tokens in, 250 tokens out. The output is only 24 percent of the tokens but 55 percent of the cost. You could halve your prompt and barely dent the bill. Trim the response and you'd feel it right away.

Workloads have also been getting heavier on both ends. The OpenRouter 100T Token Usage Study shows average prompt sizes grew roughly 4x between early 2024 and 2025, from about 1,500 tokens to over 6,000, as retrieval-augmented generation and tool use went mainstream. Completions grew about 3x over the same window.

The cost center moved. It used to sit almost entirely on output. Now RAG and agentic workloads stack thousands of input tokens onto every call, so the input side is a line item you can't ignore either.

Before you compare a single model, know your input-to-output ratio. It's the biggest driver of what you'll pay, and it decides which model is cheapest for you.

Lever 2: Model choice, and why the sticker price lies

Because input and output are priced separately, the cheapest model depends entirely on your workload shape. The ranking flips.

Picture two models:

Model A looks cheaper. It's lower on input, and its headline number is smaller. But run the token math:

• Input-heavy RAG (6,000 in, 500 out): Model A costs $0.020 per request, Model B costs $0.039. Model A wins by nearly 2x.
• Output-heavy generation (500 in, 6,000 out): Model A costs $0.061 per request, Model B costs $0.039. Model B wins, because Model A's expensive output price now dominates.

Same two models, opposite conclusions. The only thing that changed was the shape of the work.

This is why "what's the cheapest LLM?" is the wrong question. There are more than a thousand models on the market, and their input and output prices span more than two orders of magnitude. The right question is narrower: given my input-to-output ratio and my quality bar, which model has the lowest total cost? That's a lookup you should never do by hand, and it's the first thing our calculator does for you.

Lever 3: Prompt caching, real savings but not the number you were sold

Prompt caching is the most oversold lever on this list.

The idea is sound. If the front of your prompt is identical across requests, say a big system prompt, a fixed instruction block, or a set of few-shot examples, the provider can cache it and charge you a steep discount, often 50 to 90 percent off, on that portion for repeat calls. For a workload with a large, stable prefix, that adds up fast.

But hit-rate claims like "90 percent cache hits!" assume a best case you may never reach. Two things govern your real hit rate:

• How concentrated your traffic is. A cached prefix only stays warm for a short window, often about 5 minutes and sometimes up to an hour. If your next matching request doesn't land inside that window, it's a miss. Low volume means cold caches.
• How many distinct prompts you run. Traffic splits across every distinct prefix. One shared system prompt warms up fast. Fifty per-tenant templates each warm up 50 times slower, and your effective hit rate collapses.

Do the math, and a workload doing a few thousand requests a day spread across a couple dozen prompt variants can land at roughly a 15 percent hit rate on a 5-minute window, not the 90 percent on the brochure. Caching isn't broken there. It's behaving the way it always does, as a function of your traffic rather than a discount you flip on.

So size caching before you count on it. Consolidate prompt variants, concentrate traffic, and reach for the longer window where the provider offers it. Then check the number against your real volume.

We go deeper in Prompt Caching: Why Your Hit Rate Won't Be 97%, a closer look at what caching will save you.

Lever 4: Batch processing, the closest thing to free money

If a request doesn't need an answer this second, batch processing is the highest-return lever you have. Providers offer roughly 50 percent off for work you're willing to submit asynchronously and collect within a turnaround window, usually up to 24 hours.

Fifty percent off, for accepting a delay. There's almost nothing else in software where you halve the cost by changing nothing but the SLA.

The catch is the SLA itself. Batch fits anything offline:

• Document summarization and enrichment
• Structured extraction over large record sets
• Nightly evals and test suites
• Report generation and back-office pipelines

And it's a non-starter for anything a human is waiting on, like live chat, interactive assistants, or anything inside a request-and-response UI. That's the whole trade: latency for money.

The mistake we see most often is forgetting batch exists for the half of the workload that could use it. Plenty of "real-time" AI systems have a big offline tail, the reindexing job or the overnight classification pass, paying full price.

More on where the discount is real and where it isn't in LLM Batch Processing: When the 50% Discount Is Real.

The multiplier you forgot: traffic spikes

Traffic is what turns a manageable bill into a budget emergency.

Every lever above scales with traffic. So does every mistake. A workload that costs $12,000 a month at normal load doesn't cost $12,000 when a launch drives a 10x spike. It costs $120,000, and it does it in the exact week you have the least attention to spare.

The savings compound in your favor here. The gap between an optimized deployment and a naive one grows wider as traffic climbs, so the time to model the tail is before the spike. Ask now: what does this bill look like at 3x? At 10x? If you don't know, you're hoping rather than budgeting.

Model your bill before finance does

We built a free LLM Cost Calculator that turns this whole guide into a one-minute exercise. Enter your traffic and workload shape, pick from more than a thousand models, toggle caching and batch, and watch your monthly cost and your spike scenarios update live. There's no sign-up and no sales call, just your numbers.

Put it together: model your own bill

Here's the full method, in order:

• Nail your workload shape. Pull average input and output tokens per request from your provider's usage dashboard. This one ratio drives everything downstream.
• Pick the model for that shape, the one with the lowest total cost at your ratio, not the smallest sticker price.
• Size your caching honestly against your real traffic and prompt diversity. Treat the hit rate as an estimate to verify, not a discount to assume.
• Send every async-tolerant workload to batch. Find the offline tail and cut it in half.
• Model the spike. Know your bill at 3x and 10x before it happens.

Do this on a spreadsheet and it's an afternoon. Do it in the LLM Cost Calculator and it's a minute, with live charts, a model comparison across every provider, and a plain-English recommendation at the top of the report.

How TrueFoundry turns the plan into production

Modeling the savings is step one. Capturing them across dozens of models and teams, without rewriting your app every time pricing shifts, is the harder part.

That's what the TrueFoundry AI Gateway is for. One OpenAI-compatible endpoint reaches more than a thousand models across every major provider, so switching to the cheaper model for your workload shape is a config change, not a migration. Built-in cost controls, per-team usage tracking, rate limits, and fallbacks mean the savings you modeled here don't erode in production. You can see where every dollar goes and govern it.

The calculator tells you what to do. The gateway is how you do it once and keep doing it as models and prices change underneath you.

FAQ

What's the single biggest driver of LLM cost?

Your workload shape, meaning the ratio of input tokens to output tokens per request, combined with the model you pick for that shape. It moves the bill far more than the per-token sticker price you'd compare on a pricing page.

Why is my LLM bill so much higher than I estimated?

The usual culprit is estimating from price-per-token instead of real token math. Output tokens cost 3 to 5 times more than input, retrieval and tool use inflate the input far beyond a bare prompt, and traffic spikes multiply everything. Model your actual input and output tokens and your traffic to close the gap.

Does prompt caching actually save money?

Yes, but only as much as your traffic supports. Caching discounts a stable prompt prefix on repeat requests, so the savings depend on how concentrated your traffic is and how many distinct prompts you run. High volume on a few shared prompts saves a lot. Low volume spread across many prompts saves little. Size it against your real numbers before you count on it.

When should I use batch processing?

Any time the result can wait. Batch typically cuts cost by about 50 percent in exchange for asynchronous turnaround, often up to 24 hours, which makes it ideal for summarization, extraction, evals, and offline pipelines, and unsuitable for live, interactive workloads.

How do I compare LLM models on cost?

Not by sticker price. Compute total cost per request, which is (input tokens × input price) + (output tokens × output price), at your specific workload shape, then compare. The cheapest model for an input-heavy RAG app is often not the cheapest for an output-heavy generation app. The LLM Cost Calculator does this across more than a thousand models automatically.

The bottom line: the number on the pricing page is the one you should trust least. Your bill is set by your workload shape, the model you choose for it, how well you cache, and how much you batch. Every one of those is yours to control. So stop estimating and model it in about a minute.