Prompt Caching and Context Compaction: Where to Look for LLM Cost Savings

The long-context paradox: why massive token windows create operational bottlenecks

Between 2024 and 2026, frontier model providers engaged in a fiercely competitive race to expand context window limits, scaling available prompt capacities from 32,000 tokens to 128,000, 200,000, and even beyond one million tokens. This capability was widely celebrated as the end of chunking constraints: entire software codebases, legal briefs, and corporate documentation repositories could ostensibly be dumped into a single prompt without preprocessing.

However, as engineering teams deployed these massive contexts into real-world production systems—especially autonomous multi-agent loops and conversational interfaces—they collided with an uncomfortable architectural reality known as the long-context paradox. While models can technically ingest hundreds of thousands of tokens, the operational, temporal, and financial costs of doing so naively scale with unforgiving linearity.

The primary operational casualty is Time-to-First-Token (TTFT). In autoregressive transformer architectures, processing the input prompt (the prefill phase) requires a forward pass across every single input token simultaneously. While generation (the decode phase) proceeds token by token, prefilling a 100,000-token prompt across high-end GPUs still requires seconds of uninterrupted Tensor Core computation. In an interactive customer service agent or a developer terminal harness, introducing a four-to-six-second delay before the model even begins streaming its response degrades user experience and destroys conversational fluidity.

The second casualty is financial sustainability. In an autonomous agent workflow where an agent executes a multi-turn task—for example, diagnosing a cloud infrastructure issue across fifteen tool calls—each subsequent turn re-submits the entire cumulative history of the conversation. Turn 1 sends 10,000 tokens; turn 5 sends 35,000 tokens; turn 15 sends 120,000 tokens. By the conclusion of the session, the system has billed over one million cumulative input tokens for what was fundamentally a fifteen-step diagnostic script.

References: OpenAI — API Model Pricing and Enterprise Token EconomicsvLLM Project — High-Throughput and Memory-Efficient LLM Serving with PagedAttention

The physics of the KV cache: memory bandwidth and attention tensors

To comprehend why re-transmitting prompt prefixes is computationally wasteful, one must examine the physical memory mechanics of transformer inference. During the forward pass of a transformer model, each attention head projects the input token embeddings into Query (Q), Key (K), and Value (V) vector spaces.

In the prefill phase, the attention mechanism calculates the scaled dot-product of Query and Key vectors across all tokens to determine contextual attention weights, subsequently multiplying them by the Value vectors. For subsequent autoregressive decoding steps, the model only generates one new Query vector for the current token being predicted. However, to evaluate that new token's attention against all historical tokens in the sequence, the model requires the Key and Value representations of every single preceding token.

Rather than recomputing these past Key and Value vectors on every generated token, the inference engine caches them in high-bandwidth GPU memory (HBM). This in-memory tensor structure is known as the Key-Value (KV) Cache. The memory footprint of the KV cache is governed by a strict physical formula: `Memory = 2 * n_layers * n_heads * d_head * n_tokens * bytes_per_element`.

For a leading 70B parameter model operating in 16-bit precision (FP16/BF16) with 80 layers and Grouped-Query Attention (GQA) using 8 KV heads with a head dimension of 128: each token consumed in context requires approximately 160 kilobytes of dedicated GPU high-bandwidth memory. A 100,000-token context window demands 16 gigabytes of raw VRAM solely to store the KV cache for a single concurrent user session—before accounting for the 140 gigabytes required to hold the model weights themselves.

When an inference request arrives without caching mechanisms enabled, the GPU must re-read all model weights from memory and re-execute matrix multiplications across all 100,000 tokens simply to recreate the exact same KV cache tensors that were computed on the previous conversational turn. This memory bandwidth saturation is the root cause of high TTFT latency and elevated server hosting expenses.

References: vLLM Project — High-Throughput and Memory-Efficient LLM Serving with PagedAttentionNVIDIA TensorRT-LLM — Optimized Deep Learning Inference Architecture

Prefix caching mechanics: Radix Trees and hardware block reuse

The engineering solution to redundant KV computation is Prefix Caching (also referred to as Prompt Caching). In modern inference serving systems—epitomized by open-source engines like vLLM and TensorRT-LLM, as well as managed cloud APIs from Anthropic and Google—the inference engine intercepts incoming prompts and compares their token prefix against previously evaluated sequences.

In systems implementing PagedAttention (such as vLLM), the physical KV cache is partitioned into fixed-size contiguous memory blocks (typically holding 16 or 32 tokens per block), analogous to virtual memory paging in operating systems. A hierarchical data structure known as a Radix Tree indexes these memory blocks by their exact sequence of token identifiers.

When an agent sends a new request where the first 50,000 tokens—consisting of the system prompt, tool schemas, and earlier conversation turns—match an existing branch in the Radix Tree, the engine completely bypasses the prefill phase for those 50,000 tokens. Instead of executing trillions of FLOPs on GPU Tensor Cores, the scheduler simply increments the reference counters on the physical memory blocks already resident in VRAM and points the active sequence's attention table directly to the existing cache.

The operational impact of this architectural shift is dramatic. The prefill computation for the cached prefix drops from seconds to zero; the engine immediately proceeds to prefill only the newly appended tokens (the latest user message or tool result). TTFT latency drops by up to 85%, and commercial API providers pass these hardware savings directly to developers in the form of 75% to 90% discounts on cached input tokens.

References: vLLM Project — High-Throughput and Memory-Efficient LLM Serving with PagedAttentionOpenAI — API Model Pricing and Enterprise Token Economics

Structuring prompts for maximum cache hit rates: architectural guidelines

While prefix caching provides enormous theoretical savings, achieving high cache hit rates in production requires disciplined software design. Because Radix Trees match prefixes sequentially from the very first token, any variation at the beginning of a prompt invalidates the cache for all subsequent tokens.

A common anti-pattern observed in poorly architected agent systems is injecting dynamic, ephemeral data into the top of the system prompt. For instance, prepending the current timestamp (`Current time: 2026-09-22 14:02:18`), a randomized user session ID, or dynamic weather data to line 1 of the system prompt guarantees that every single API request has a unique prefix. This single flaw reduces the cache hit rate to exactly zero percent across all users and all conversational turns.

High-performance prompt architecture enforces a strict immutable-to-mutable hierarchy. Prompts must be organized in order of stability: immutable system core instructions first, followed by static tool definitions, followed by reusable retrieved context documents, followed by cumulative conversation history, and terminating with the latest dynamic user input.

References: Anthropic — Introducing the Model Context ProtocolGoogle Search Central — AI features and your website

Context compaction strategies: pruning agentic observation bloat

While prompt caching optimizes the cost of repeated tokens, autonomous agents operating over extended workflows inevitably encounter context window degradation. Even with prefix caching, allowing an agent's context to expand indefinitely to 100,000+ tokens causes severe attention dilution, increases decoding latency, and risks hitting hard token quotas.

The primary driver of context bloat in agentic architectures is what we identify as Observation Bloat. When an agent calls an MCP tool—such as querying a PostgreSQL database, scanning an S3 bucket, or reading an HTTP endpoint—the tool often returns hundreds of lines of raw JSON containing verbose database column metadata, tracking headers, and null fields. The agent typically only needs two values from that entire payload to proceed, yet the entire raw blob remains permanently embedded in the conversation history.

Production agent harnesses implement deterministic Observation Pruning. Once an agent has consumed a tool result and successfully executed its subsequent reasoning step, the harness replaces the raw, verbose tool payload in historical turns with a compact, structured semantic receipt. For example, a 3,000-token raw JSON array of 50 customer invoices is compressed into: `[Tool Result Pruned: Retrieved 50 invoices. Target Invoice #8841 validated as Paid ($4,250.00).]`.

This compaction reduces the token footprint of historical turns by 80% to 95% without compromising the model's forward reasoning capabilities. The model retains the precise factual conclusion it reached, while the transient scaffolding that produced that conclusion is safely discarded from active VRAM.

References: Anthropic — Introducing the Model Context ProtocolvLLM Project — High-Throughput and Memory-Efficient LLM Serving with PagedAttention

Hierarchical state compaction: the Sliding Window with Pinned Anchors pattern

When an autonomous agent must operate across long-running, multi-day or multi-stage workflows—such as SRE incident remediation, automated codebase refactoring, or multi-chapter technical authoring—simple observation pruning is insufficient. The architecture must adopt Hierarchical State Compaction.

The most resilient pattern observed in enterprise agent engineering is the Sliding Window with Pinned Anchors. In this architecture, the context is partitioned into three distinct zones: the Pinned Header, the Compressed Epoch Summary, and the Active Sliding Window.

The Pinned Header (500 to 2,000 tokens) contains the unalterable system prompt, high-level business goals, and security boundaries. The Active Sliding Window retains the most recent three to five conversational turns (e.g., the last 10,000 tokens) with full granular detail, ensuring the model maintains immediate conversational flow and short-term working memory.

When the total token count approaches an established operational ceiling (e.g., 32,000 tokens), the harness triggers an asynchronous compaction routine. A specialized distillation agent or internal pipeline analyzes the oldest turns in the sliding window, extracts key decisions, resolved blocker states, and updated entity facts, and appends them to the Compressed Epoch Summary. The raw historical turns are then evacuated from the active prompt and archived to durable relational storage (such as PostgreSQL).

This tripartite structure ensures that the total active context window never exceeds a flat, predictable token boundary, eliminating runaway API bills and keeping inference latencies strictly bounded throughout the entire operational lifecycle.

1. Step 1: Monitor active session token volume using real-time tokenizer counts before each inference dispatch.
2. Step 2: When token threshold (e.g., 75% of operational budget) is exceeded, initiate the compaction pipeline.
3. Step 3: Preserve the Pinned Header (Layer 1 & Layer 2) completely untouched to maintain prefix cache continuity.
4. Step 4: Execute structured observation pruning on historical tool turns, stripping redundant JSON schemas and raw network payloads.
5. Step 5: Summarize older conversational turns into a verified 'State Delta' document and commit raw historical logs to durable database storage.
6. Step 6: Dispatch the compacted prompt, restoring low-latency sub-second generation while preserving full task provenance.

References: vLLM Project — High-Throughput and Memory-Efficient LLM Serving with PagedAttentionGoogle Search Central — AI features and your websiteSchema.org — Structured Data Vocabulary Standard

Production checklist: FinOps and context hygiene for engineering leaders

Before deploying high-frequency agentic systems or large-context applications into production environments, engineering teams must validate their infrastructure against strict context hygiene and FinOps controls.

First, audit prompt serialization order. Verify that dynamic timestamps, random nonces, and user-specific IDs are never injected above static system instructions and tool manifests. Inspect inference telemetry to confirm prefix cache hit rates exceed 70% in multi-turn workflows.

Second, implement strict tool return payload sanitization. Establish middleware filters in your Model Context Protocol (MCP) clients that strip extraneous metadata, paginate large collections, and enforce maximum byte boundaries on tool output before injecting data into the agent's context.

Third, model your exact token economics and compare managed API tier costs against dedicated GPU cluster hosting. Test your parameters on the Cendar Lab Inference FinOps Calculator (/tools/inference-calculator) and audit your domain's agent accessibility with the AEO Checker (/tools/aeo-checker).

Fourth, establish automated regression alarms on Time-to-First-Token (TTFT). Alert engineering teams whenever p95 TTFT exceeds 1,500 milliseconds, identifying whether context bloat, cold cache misses, or GPU queue congestion is causing operational friction.

By treating context not as a disposable infinite bucket, but as an expensive, high-bandwidth computational memory asset, engineering teams build agent systems that are blisteringly fast, commercially viable, and architecturally resilient.

References: OpenAI — API Model Pricing and Enterprise Token EconomicsvLLM Project — High-Throughput and Memory-Efficient LLM Serving with PagedAttentionNVIDIA TensorRT-LLM — Optimized Deep Learning Inference Architecture