TL;DR: No single technique gets you to 80% savings on LLM inference cost — the number comes from stacking optimizations across three independent layers: the model itself (quantization, distillation), the serving system (continuous batching, KV cache management, speculative decoding), and the application (prompt caching, semantic caching, model routing). Each layer alone typically yields 20–50%; combined, teams report 60–90% reductions, but only when the workload mix actually fits the techniques. A bill that's 80% smaller on paper and 20% smaller in practice usually means one layer was implemented without checking whether the traffic pattern supported it.

Every LLM inference cost conversation eventually arrives at the same question: "which one thing should we do first?" The honest answer is that there isn't one thing — cost optimization here is genuinely layered, and treating it as a single lever (usually "switch to a cheaper model") leaves most of the available savings on the table. Here's the layer-by-layer breakdown, in the order that tends to produce the best return for the effort.

The three layers, and why order matters

LLM inference optimization splits cleanly into three layers: model-level (quantization, pruning, distillation — changes to the model artifact itself), system-level (continuous batching, PagedAttention-style memory management, speculative decoding — changes to how the serving infrastructure runs the model), and application-level (prompt caching, semantic caching, model routing — changes to what you send the model and which model you send it to) (Morph, 2026).

The order matters because application-level changes are usually the cheapest to implement and the fastest to show results, while model-level changes (re-quantizing, retraining, distilling) require the most engineering investment and the most validation before you trust them in production. If you're starting from zero, work top-down through application, then system, then model — not because model-level techniques matter less, but because you want to have already captured the easy wins before spending weeks on the hard ones.

Application layer: caching and routing (start here)

Prompt / prefix caching

This is usually the single highest-leverage change available, and it's also the one requiring the least engineering: cache the KV state for a shared prompt prefix so that a new request sharing that prefix skips recomputing it entirely. Anthropic's cached-token pricing is roughly $0.30/M tokens versus $3.00/M for fresh processing — a 90% discount on the cached portion — while OpenAI offers a 50% discount on cached tokens (Digital Applied, 2026).

This technique's payoff is entirely a function of your prompt structure. If your system prompt, few-shot examples, or retrieved context is large and stable across many requests (a common RAG pattern, a coding assistant with a large fixed system prompt, an agent with a static tool-definition block), prefix caching can eliminate the majority of your prompt-processing cost. If every request has a genuinely unique prompt with no shared prefix, this technique does nothing for you — check your actual prompt structure before assuming this will help.

Structuring for cache hits: put stable content (system prompt, tool definitions, static context) at the start of the prompt and variable content (the user's specific query, dynamically retrieved documents) at the end. Providers cache based on prefix matching, so anything that varies needs to come after everything that doesn't, or you break the cache on every request.

Semantic caching

Distinct from prefix caching: semantic caching stores full responses keyed by embedding similarity, so a repeated or near-duplicate query (not necessarily an identical prefix) can return a cached answer without a model call at all. This works well for FAQ-style or support-deflection workloads with genuinely repetitive query patterns, and does close to nothing for workloads where every query is meaningfully unique (most agentic and coding-assistant traffic). It's worth noting this is a different lever than choosing a retrieval architecture in the first place — if your cost problem is actually about how much context a RAG architecture is stuffing into every prompt, the fix is upstream of caching, and often the more fundamental question is whether fine-tuning vs RAG vs prompting was the right call for the task at all.

Model routing

Route requests to the cheapest model capable of handling them, rather than sending everything to your most capable (and most expensive) model by default. A classifier or heuristic — often a much smaller, cheaper model — decides whether a query needs the frontier model or can be handled by a smaller one. This requires investment in an eval set that actually measures whether the smaller model's answers are acceptable for the queries you're routing to it; skipping that step is how routing quietly degrades output quality while looking like a clean cost win on the dashboard. A properly built LLM eval pipeline with golden datasets and CI gates is what makes that measurement trustworthy rather than a one-off spot check.

Combined, the application layer (provider KV cache, an explicit prompt-cache strategy for large stable prefixes, and semantic caching for repetitive queries) can realistically cut total inference cost by 70–90% for workloads that fit the pattern — but that range assumes your traffic actually has the repetition and shared-prefix structure these techniques exploit (Digital Applied, 2026).

System layer: how the serving infrastructure runs the model

If you're calling a hosted API (OpenAI, Anthropic, etc.), this layer is mostly the provider's problem — you benefit from their batching and caching infrastructure automatically. If you're self-hosting (vLLM, TGI, SGLang, or similar), this layer is where a meaningful share of your cost sits.

Continuous batching. Static batching waits for a full batch before processing; continuous batching dynamically inserts new requests into a running batch as older ones complete, keeping GPU utilization high instead of leaving it idle between batch cycles. Reported throughput improvements range from 3–10x over naive batching, with one production case measuring a 23x aggregate throughput improvement after enabling it (Hakia, 2026). If you're self-hosting without continuous batching enabled, this is very likely your highest-leverage remaining lever.

KV cache memory management. The KV cache — the per-token attention state the model needs to avoid recomputing previous tokens — grows linearly with sequence length and is often the actual memory bottleneck limiting how many concurrent requests a GPU can serve, not the model weights themselves. PagedAttention-style memory management (treating KV cache like paged virtual memory rather than requiring contiguous allocation) reduces memory fragmentation and lets more requests share the same GPU (arXiv survey, 2026). Emerging techniques compress the KV cache itself — recent work claims 3-bit KV cache compression with negligible measured accuracy loss, a roughly 6x memory reduction over standard precision — which is worth tracking if you're memory-constrained rather than compute-constrained.

Speculative decoding. A small, fast "draft" model proposes several tokens ahead, and the larger target model verifies them in a single forward pass instead of generating token-by-token — when the draft model's guesses are accepted, you get multiple tokens for roughly the cost of one forward pass. This helps latency more directly than raw cost, but lower latency per request often translates to better GPU utilization at the same concurrency level, which does show up in your cost per token. It adds real complexity (maintaining a draft model, managing acceptance-rate tuning) and composes awkwardly with other techniques — serving stacks report real engineering effort to make speculative decoding work cleanly alongside disaggregated prefill/decode and KV-cache-aware routing simultaneously (SGLang roadmap discussion, 2026). Treat this as a layer-two-or-later optimization, not a first move.

Model layer: changing the model itself

This is the highest-effort, highest-validation-burden layer, and the one to tackle after you've captured the application and system wins.

Quantization. Reducing weight precision from FP16 to INT8 or INT4 cuts memory footprint 2–4x and typically reduces inference cost by roughly half, while retaining 95–99% of original task accuracy for most workloads (Hakia, 2026). The accuracy retention figure is workload-dependent — always validate on your own eval set rather than trusting a vendor's general accuracy claim, since quantization degradation shows up disproportionately on tasks requiring precise numerical reasoning or exact-format output.

Distillation. Training a smaller model to mimic a larger one's outputs on your specific task distribution can produce a model that's both cheaper to serve and, for a narrow enough task, comparably accurate to the larger original — but it requires investing in a training pipeline and a representative dataset, and it locks you into re-running that pipeline whenever the task distribution shifts meaningfully. This is worth it for a high-volume, narrow, stable task (classification, extraction, a fixed-format generation task) and rarely worth it for open-ended generation where "the task" keeps changing.

Pruning. Removing redundant weights or attention heads with minimal accuracy impact — less commonly deployed in production than quantization because the tooling is less mature and the accuracy/compression tradeoff is harder to validate reliably, but worth watching as the tooling matures.

Putting the layers together: a realistic sequencing plan

  1. Measure first. Before touching anything, get a real breakdown of where your token spend actually goes — by endpoint, by prompt type, by whether requests share a prefix. Optimizing blind is how teams implement semantic caching for a workload with no query repetition and wonder why the bill didn't move.
  2. Application layer, week 1–2. Restructure prompts so stable content is a shared prefix, turn on provider prompt caching, add semantic caching if your query distribution has real repetition. This is the highest-ROI, lowest-risk starting point for nearly every team.
  3. Add model routing, week 2–4, but only after building the eval set that tells you which query types tolerate a cheaper model. Route based on measured accuracy tolerance, not intuition about "easy" vs. "hard" queries.
  4. System layer, if self-hosting, week 3–6. Turn on continuous batching if it isn't already; audit KV cache memory management; evaluate whether your concurrency is memory-bound or compute-bound before choosing further optimizations.
  5. Model layer, only once 1–4 are in place and you still need more. Quantize with a validation gate against your own eval set, not a general benchmark. Reserve distillation for narrow, high-volume, stable tasks.

Sources: Morph — LLM Inference Optimization: Cut Cost & Latency at Every Layer, Hakia — LLM Inference Optimization Techniques, GMI Cloud — Cutting LLM Inference Costs in 2026, Digital Applied — Prompt Caching in 2026, arXiv — Survey on System-Aware KV Cache Optimization, Red Hat Developer — KV Cache Aware Routing with llm-d.

Syslabs' engineering team builds and tunes production LLM serving pipelines with this kind of layered cost discipline for clients running AI features at scale.