Your long-context agent bill is a memory-layout problem

Update, October 9, 2026: the worked example now uses the current Sonnet 5.5 cache-read price of $0.10 per million (it was $0.20 when first published).

Two numbers decide what a long-running agent costs and whether it can run at all: how many bytes the KV cache needs per session, and how much of each prompt you pay full price for again. Both moved this year.

1. KV bytes, not the context spec, set the real limit

A 1M-token context window used to be a spec-sheet number. On DeepSeek V3.2 it meant roughly 84 GiB of KV cache per sequence (bf16 estimate). One long agent session could push an entire batch out of HBM.

DeepSeek-V4 needs about 9.6 GiB for the same 1M context, per vLLM's analysis. Hugging Face puts it near 2% of a standard grouped-query-attention cache. DeepSeek-V4-Pro uses about 10% of V3.2's KV memory and 27% of its single-token FLOPs; V4-Flash uses about 7% of the KV cache.

What changed in the design:

• Compressed Sparse Attention: 4x KV compression, with a lightning indexer (running in FP4) picking top-k blocks.

• Heavily Compressed Attention: 128x compression.

• A sliding-window branch keeps the most recent tokens exact.

• FP8 for most KV entries, BF16 only for the RoPE dimensions.

The second-order effect is on serving. An engine now manages several cache types with different compression ratios. vLLM folds five of them into three page-size buckets over 256-token blocks so fragmentation doesn't eat the savings.

2. Most of an agent's prompt doesn't change between turns

A recent arXiv paper on stateful inference (2605.26289) reports that 85 to 95 percent of the prompt is unchanged between turns in agent loops. If that holds for your workload, resending the full history every turn means paying again for tokens you already processed.

Published prompt-caching prices show the size of that gap. For Claude Sonnet 5.5, Anthropic lists $2 per million input tokens and $0.10 per million for cache reads (writes are $2.50). That is after an October 2026 price cut; it was $0.20 when I first wrote this. A cache read now costs a twentieth of fresh input.

Illustrative arithmetic, from the published prices and not a measurement: take a 100K-token prompt where 85% is unchanged and cached.

• Uncached: 100,000 tokens at \(2/MTok = \)0.200

• Cached: 85,000 at \(0.10/MTok (\)0.0085) plus 15,000 at \(2/MTok (\)0.030) = $0.0385

That is about 81% lower input cost on that turn. It ignores the one-time cache write and any output tokens, and your real unchanged fraction will differ.

3. How the two fit together

The first number is about whether a long session fits in memory. The second is about whether you pay for it repeatedly. Both reward the same design habit: keep the stable part of the context stable, and treat cache layout as part of the architecture, not an afterthought.

Anthropic also states that Sonnet 5.5 uses fewer tool calls and costs up to 30% less per task. That is the vendor's claim and I haven't verified it. It is still the right unit: cost per solved task, not price per token.

What I'd do with this

• Model KV bytes per session before comparing parameter counts.

• Measure what fraction of your prompt is unchanged between turns. That fraction is your caching upside.

• Compare models on cost per solved task on your own workload.

Sources

• Hugging Face, DeepSeek-V4: https://huggingface.co/blog/deepseekv4

• vLLM, DeepSeek-V4 serving: https://vllm.ai/blog/2026-04-24-deepseek-v4

• arXiv 2605.26289 (stateful inference)

• Anthropic, Claude Sonnet 5.5 pricing page

For teams running long-context agents: is KV memory or repeated prompt cost your binding constraint today?

---

Work with me: I take 1:1 calls on mobile architecture reviews, mobile and AI questions, and startup tech consults. Book a slot on Topmate: topmate.io/sairamkotha