This article follows the post: what broke, what Scion changes, and what a team that is not Anthropic can take from it.

How to read it: every claim about Scion comes from Anthropic's engineering post "Redesigning source control at Anthropic", published October 13, 2026, which I read on October 14. I have not used Scion and have no inside knowledge. The arithmetic sections are mine and are labeled. Where the post was silent, I say so.

The numbers behind it

The post gives these figures for Anthropic's monorepo:

• Merges went from around 2,500 code changes per day in May 2026 to over 30,000 per day in September.
• The history of the main branch alone is almost 100 GB, and a compressed working copy is over 25 GB.
• CI performs on the order of 25 million code checkouts per day, before developer environments, agent sandboxes and research jobs.
• Fetched naively, that would be over 600 PB of git traffic per day.

The post's first line says that as merges per day grew more than tenfold, source control became the biggest development bottleneck.

My arithmetic on those figures: 30,000 merges a day is about one every 2.9 seconds, around the clock. 25 million checkouts a day is about 290 per second. And 600 PB divided by 25 million is about 24 GB per checkout, which lines up with the 25 GB working copy. So the 600 PB figure is just "every checkout downloads the whole repo."

Why git broke

The post's core diagnosis is a mismatch. Git is built for distributed, offline work: every machine holds the history and can work alone. Anthropic was using it as a centralized system with one authoritative copy, whose clients are nearly always online agents and jobs. The post says the properties that make git excellent for distributed, offline work were actively undermining that use.

The team had piled up the standard workarounds: shallow clones, sparse checkouts, promisor remotes, internal git mirrors and blob-storage seeds. At its peak Anthropic ran several hundred internal git mirrors. Three things gave out:

• Seeds drifted further from the tip of main between refreshes.
• Diff and blame on partial clones got harder to keep working as the merge rate rose.
• Mirrors stopped scaling once the repo outgrew their disks or memory, and became harder to operate as the merge rate increased.

None of this means git is slow. It means a design for a thousand offline laptops met thirty thousand merges a day from machines that are always connected.

What Scion does differently

The post lists six design principles:

1. Clients are always online. A client talks to a reachable server and fetches data when it needs it.
2. Client-server asymmetry. The server handles history, blame, commits, merges and rebases. The client keeps a bounded local cache and lays out the working copy.
3. Source control as a service. Queries are exposed through an API, so many jobs need no local repository. Automated bisect and revert pipelines depend on this.
4. Horizontal scalability. No full copy of the repo exists anywhere, even on the server, and there are no packfiles or repo maintenance.
5. Cheap, lock-free commits. Commits, rebases and merges happen on the server, so there is nothing to push. Refs are discouraged except for main and release branches.
6. Optional, sparse working copies. CI can ask the server for diffs, and many writes skip checkout entirely.

The data model is a changeset, an entry in a write-ahead log, created on the server. It lists changed files relative to its first parent, with content-addressed pointers to compressed blobs. Refs are a compare-and-swap on a single Spanner row. Blame, path history and commit-graph skip lists are derived data, computed from parent values, so the internals can change without client changes.

A small fleet of servers answers all the APIs. Because everything is content-addressed, any server can cache it without coordination. The client fetches only the trees and blobs it wants, starting at a root tree and recursing, so there is no multi-gigabyte packfile and transfers are resumable.

Delivery is tiered: a scion-daemon on each node, a regional scion-cache, and the authoritative scion-server that reads from Spanner or cloud storage on a miss. The post says tiering is strictly enforced so a failing regional cache cannot overwhelm upstream servers.

What Anthropic reports

All figures below are the post's own claims, not independent measurements:

• The Scion infrastructure footprint is 200x smaller than the previous git mirroring infrastructure, in CPU and memory.
• CI and research jobs check out code two orders of magnitude faster, in seconds rather than minutes.
• CI alone realized a 25% CPU capacity win.
• Each tier keeps a cache hit rate of over 99%.
• A team of seven people built Scion and the code review tool Antique and migrated to both in five months.
• Claude Mythos Preview rewrote the git-to-Scion ingestion pipeline, producing a version that used 20x less memory and ran 4x faster.

The post does not describe the cutover plan or downtime, so I can say nothing about how the migration was done.

The part most people skip

Two paragraphs in the post are about people and process, not storage.

Human-scale practices stopped fitting. The post says Anthropic has not been able to effectively replicate the industry standard of a global repository lock on the merge queue. It has invested in post-merge and pre-deploy automated testing instead, and says Claude can typically fix conflicts faster than humans can even identify them. Reviewers now examine the intent of a change instead of reading every line, and the code review tool can express complex semantic constraints on approvals.

The engineer's job moved toward saying no. The post says the role of the engineers was to tell Claude "no," and gives two examples: multiple systems that tried to brute-force looking up when a change merged to main, and one review that was overly defensive about trusting another internal service.

Who should not copy this

The post is direct about the limits. Scion is "by no means a suitable replacement for git for general use," and it fits an organization with either tens of thousands of engineers or swarms of agents prolifically writing code. The post says Anthropic's ability to spend tokens for internal use far exceeds what any other business would realistically want to spend. And because Scion diverges from git, adopting it means replacing code review, code search and CI too, which are highly bespoke inside Anthropic.

The post does not announce a release. It says it is difficult to offer Scion in a form that would be broadly useful, and that the most impactful thing Anthropic can do is share its thinking. It names three directions for the wider ecosystem: git servers that do not mirror client behavior, server-side RPCs on git platforms, and a redesigned negotiation protocol where sparse and shallow features compose well.

What a smaller team can take from it

This section is my opinion, not the post's.

1. Count your own checkouts. Most teams have never added up how many times per day CI, agents and sandboxes fetch the repo. At agent scale that number, not developer headcount, is what breaks things.
2. Treat source control as an API for machines. If a job only needs to know what changed, asking a server beats cloning.
3. Look for human-scale locks. Merge queues, serialized deploys and manual review gates all assume a human pace. Anthropic's own post says it could not make a global merge-queue lock work at its volume.
4. Do not rebuild git. Anthropic needed this at 30,000 merges a day with a seven-person team and its own models doing the work. Try the sparse and shallow tools git already has first.

What I could not verify

I read one primary source and did not test Scion or run any of the numbers. Every performance figure is Anthropic's claim. The post does not describe the cutover plan, downtime, or whether the 600 PB figure was ever actually transferred rather than being a calculation. Several news write-ups say Anthropic will not open-source Scion or give a different reading of the post. I used only the post's own wording. Details may change.

Source, accessed October 14, 2026: Anthropic Engineering, "Redesigning source control at Anthropic" (anthropic.com/engineering/redesigning-source-control).

If your agents started merging ten times more code next quarter, which part of your pipeline would break first?

#AI #Git #SoftwareEngineering #AIAgents

Work with me: I take 1:1 calls on mobile architecture reviews, mobile and AI questions, and startup tech consults. Book a slot on Topmate: topmate.io/sairamkotha

This article follows that cost through the documentation: what Anthropic says its newer tokenizer does, what it does to the price of the same text, why counts differ between providers, and how to measure your own.

How to read it: every claim about tokenizers and prices comes from the providers' documentation, the pricing page, and one 2026 paper, which I read on October 14, 2026. The cost examples are my arithmetic. I ran no live token counts, so I have no measurement of my own. Where a page was silent, I say so.

A token is not a unit of text

A tokenizer splits text into pieces from a fixed vocabulary, and each model family has its own. Billing, rate limits and context windows are all counted in those pieces. So a token is a unit defined by one model's vocabulary, not a fixed amount of text.

A 2026 paper, "How Long Is a Piece of String?", compares ten tokenizers (Claude, DeepSeek, Gemini, Grok, GPT, Mistral, Llama 4, Reka, Jamba and Qwen) on eight kinds of text. Its findings:

• Compression for the same kind of text varies across tokenizers by roughly 20 to 50 percent, and by about 100 percent for emoji and digit strings.
• Within one tokenizer, the gap between the worst case (emoji) and the best (code and essays) is nearly tenfold.
• The common rule of thumb of 0.75 words per token is too simple. For Claude and Gemini the average is above 0.75 for the most frequent words. For DeepSeek and Mistral it is closer to 0.6, and for rare words it falls to 0.35 to 0.45.
• One long word, "antidisestablishmentarianism", takes 9 tokens on Claude and 5 on Gemini or Grok.
• Reported context limits are not comparable: equal limits in tokens hold different amounts of text.

The authors conclude that tokenization varies significantly across models and domains, which makes naive interpretation of token counts problematic.

The providers say the same in softer language. OpenAI's help page says the same text can produce different token counts depending on the model, its encoding and the language, and points to its tokenizer tool and tiktoken for exact counts. Gemini's page gives a rule of thumb of about 4 characters per token and offers a count_tokens call that counts the input only.

What Anthropic changed

Anthropic's token counting page states it directly: Claude 4.7 and later models use a newer tokenizer, and the same input text produces approximately 30 percent more tokens than on earlier models. The exact increase depends on the content and workload shape. The Opus 5.5 migration guide gives the range: roughly 1x to 1.35x as many tokens, up to about 35 percent more, varying by content.

Three details matter:

1. Billing follows the new counts. The page says usage and billing on these models reflect the new tokenizer's counts, and says not to reuse token counts measured on an older model to estimate cost or context fit.
2. The pricing page lists the split. Claude Sonnet 4.6 and earlier use the previous tokenizer. Claude 4.7 and later use the new one.
3. It reaches beyond the invoice. The migration guide tells you to give max_tokens additional headroom, including compaction triggers, and to re-test any code that estimates tokens client-side or assumes a fixed token-to-character ratio.

Anthropic does not publish the multiplier per content type on these pages. Some third-party posts report that code and structured data inflate more than prose. I did not use those figures, because I could not trace them to a primary source.

What it does to the price of the same text

The pricing page lists each model's price per million input tokens. Take the same text that costs 1 million tokens on a pre-4.7 model, and apply the documented roughly 30 percent. All figures below are my arithmetic on input tokens only.

Haiku 5.5 is shown at its under-100,000-token prompt price. Above that it is $0.50.

Read it as a monthly bill. A workload of 100 million input tokens a month on Opus 4.6 costs $500. The same text on Opus 4.7 is about 130 million tokens at the same $5, so $650. On Opus 5.5 it is 130 million at $4, so $520. On Sonnet, $300 on 4.6 becomes about $260 on Sonnet 5.5.

Two lessons:

• Opus 4.6 to 4.7 is a pure tax. The list price did not change and the same text costs about 30 percent more. This is the case the headline describes.
• Where the price was cut, the tokenizer takes back part of the cut. The cut was 33 percent on Sonnet input, but the effective saving is about 13 percent.

Because the multiplier is a range of 1.0 to 1.35, the answer depends on your content. Opus 5.5 costs more than Opus 4.6 for the same text once the multiplier passes 1.25 (5 divided by 4). At the top of the documented range, 1.35, that is $5.40 against $5, or 8 percent more. Sonnet 5.5 stays cheaper than Sonnet 4.6 across the whole documented range, because the break-even is 1.5.

Cached reads scale the same way. A cache read on Opus 4.6 and 4.7 is $0.50 per million tokens, but the same cached text is about 30 percent more tokens on 4.7, so about $0.65 for the same text. Caching lowers the price per token. It does not lower the token count.

I did not find a documented multiplier for output tokens. The pricing page says the new tokenizer produces about 30 percent more tokens for the same text, which would apply to a reply too, but I did not see an output-specific figure, so I left output out of the table.

What else counts in tokens

Anything measured in tokens changes with the tokenizer:

• Context window. A window of fixed size in tokens holds less text. At 1.3x, it holds about 77 percent as much (1 divided by 1.3).
• Rate limits. Anthropic's input and output limits are per minute in tokens, so the same traffic may use more of them. That is my inference from the limits being token-based. I did not test it.
• Compaction and truncation thresholds. If you trigger them at a token count you chose on the old model, they now fire at about 23 percent less text.
• Client-side estimates. Any code that estimates tokens as characters divided by four is now wrong in a way that depends on content.

How to measure your own

Anthropic's token counting endpoint is free. Its limits are separate from message creation, and it counts under the tokenizer of the model you name. The docs suggest counting the same request twice, once with your current model and once with the one you plan to move to.

import anthropic

client = anthropic.Anthropic()
request = dict(
    system=YOUR_SYSTEM_PROMPT,
    tools=YOUR_TOOLS,
    messages=YOUR_TYPICAL_MESSAGES,
)

for model in (CURRENT_MODEL, TARGET_MODEL):
    n = client.messages.count_tokens(model=model, **request).input_tokens
    print(model, n)

Three caveats from the same page:

• The count is an estimate and may differ by a small amount from actual usage.
• Counts may include tokens that Anthropic adds for system optimizations, and you are not billed for those.
• It does not use caching. Cache read and write pricing applies only when you create a real message.

The endpoint also counts tools, images and PDFs, so you can measure your real request, not just a text sample. It returns an error for server tools such as web search and for the MCP connector. For those, read the usage object on a real response.

What to do about it

1. Recount before you migrate. Run your real prompts through count_tokens on both models and compare. Do not reuse old counts.
2. Compare cost per task, not per token. Per-token price alone misleads in both directions, as the table shows.
3. Add headroom. Raise max_tokens and move compaction triggers, as the migration guide says.
4. Check your content type. If most of your traffic is short prose, you may be near 1.0x. If it is code or structured data, assume the top of the range until you measure.
5. Do the same on other providers. OpenAI points to tiktoken and Gemini to count_tokens. If you route between providers, one shared token budget will not hold the same amount of text on each.
6. Re-test token-based assumptions. Search your code for division by four and for hard-coded token thresholds.

What I could not verify

I have no API key in this environment, so I ran no live token counts and cannot confirm the multiplier on your workload or mine. The 30 percent and the 1x to 1.35x range are Anthropic's figures, and the table is my arithmetic on them. I did not find a documented output-token multiplier, a per-content-type breakdown from Anthropic, or whether rate limits are applied after the new counts. For other providers I read only the general token pages, not model-specific release notes, so I make no claim that OpenAI or Google changed a tokenizer recently. The paper's per-domain values are in a figure I could not read as text. Prices, models and tokenizers change, so recheck before relying on any number.

Sources, all accessed October 14, 2026: Anthropic: Token counting; Pricing; Claude Opus 5.5 migration guide (platform.claude.com/docs). OpenAI: What are tokens and how to count them (help.openai.com). Google: Understand and count tokens (ai.google.dev). Roberts, Han and Albanie, "How Long Is a Piece of String? A Brief Empirical Analysis of Tokenizers", arXiv:2601.11518.

When did you last recount a prompt after switching models, and how far was the new number from the old one?

#AI #LLM #AIEngineering #Tokens

Work with me: I take 1:1 calls on mobile architecture reviews, mobile and AI questions, and startup tech consults. Book a slot on Topmate: topmate.io/sairamkotha

This article follows one screenshot through five providers' documentation: how each counts it, what it costs in a long chat, and what the docs say about caching and image changes.

How to read it: every behavior and formula below comes from the providers' own documentation, which I read on October 13, 2026. The token counts for a 1080p screenshot use each page's formula, and the cost examples are my arithmetic with assumed chat sizes. I ran no live token counts. Where a page was silent, I say so.

How an image becomes tokens

A text model reads tokens: pieces of words produced by a tokenizer. A vision model needs the picture in the same currency, so providers cut the image into a grid of fixed-size square patches and count the patches. Anthropic's docs say Claude views images in patches instead of pixels, each a 28x28-pixel block called a visual token. OpenAI's formula starts with a patch count too, using 32x32 patches. Alibaba's Qwen3 models use a 32x32-pixel block per token.

Three things follow, because these formulas depend only on pixel dimensions:

1. Cost grows with area, not file size. Doubling both sides gives about four times the patches. On Claude, a 512x512 image is 361 tokens and a 1024x1024 image is 1,369, which is 3.8 times more.

2. Content does not matter. By these formulas, a mostly white 1080p screenshot costs the same as a dense one, and a small JPEG costs the same as a large PNG with the same dimensions. File size affects request size and upload limits, not tokens.

3. There is a ceiling. Every provider has a resolution cap, and larger images are scaled down first, so cost stops growing. On the newest Claude models the cap is 4,784 tokens, at 2,576 px on the long edge. Pixels beyond it add upload size, not tokens.

Gemini counts differently, in tiles of 258 tokens, with the tile count set by the image's aspect ratio rather than its size. The details are in the next sections.

An image is billed as input tokens

OpenAI says it in one line: vision models convert image inputs into billable input tokens, and image tokens also count toward your tokens-per-minute limit. Anthropic's vision page tells you to multiply the image's token count by the model's per-token price. Alibaba's billing section says input includes text tokens and image or video tokens.

So an image costs its token count at your normal input price, every time it is sent. The interesting question is how many tokens that is, because the five providers I read count differently.

How five providers count the same 1920x1080 screenshot

• Anthropic. The image is cut into 28x28-pixel patches, and it costs the number of patches: ceil(width/28) x ceil(height/28). There are two tiers. Claude 4.7 and later models, which includes Opus 5.5, Sonnet 5.5 and Haiku 5.5, accept up to 2,576 px on the long edge and 4,784 tokens. Older models cap at 1,568 px and 1,568 tokens. The docs' own table: a 1920x1080 image costs 2,691 tokens on the high-resolution tier and 1,560 on the standard tier. The docs add that high resolution can use up to roughly three times the tokens of the standard tier.

• OpenAI. Current models use 32x32 patches. The patch count is multiplied by 1.2, and detail: high caps the image at 2,500 patches on the models I read. My arithmetic for 1920x1080: 60 x 34 = 2,040 patches, times 1.2, is 2,448 tokens. Older models such as gpt-4o use 512 px tiles instead, at 85 base tokens plus 170 per tile.

• Qwen (Alibaba Model Studio). For the Qwen3 and Qwen3-VL families, one token is a 32x32 pixel block, and the docs give the formula height x width / 1,024, plus 2. The default pixel cap is about 2,560 tokens, and a vl_high_resolution_images flag raises the cap to 16,384. My arithmetic for 1080p: about 2,040 tokens, plus 2. Qwen2.5-VL uses 28x28.

• Google Gemini. An image with both sides at or under 384 px costs 258 tokens. Larger images are cut into 768x768 tiles at 258 tokens each, with a crop unit of roughly the shorter side divided by 1.5. By that formula a 1920x1080 image is 6 tiles, or 1,548 tokens. Gemini 3 adds a media_resolution setting, but the page I read gives no per-setting counts.

• DeepSeek. Its pricing page lists vision as supported on deepseek-flash and not supported on deepseek-v4-pro. I did not find an image token formula in the pages I read.

Same pixels, four documented counts: about 1,550 to 2,690 tokens. Token price per million differs more than that across models, so check both numbers.

What an image costs by size

Tokens per image, from each provider's formula (the OpenAI, Gemini and Qwen numbers are my arithmetic):

• 512x512: Claude 361, GPT-5.x 308, Gemini 1,032, Qwen3-VL 258.

• 1024x1024: Claude 1,369, GPT-5.x 1,229, Gemini 1,032, Qwen3-VL 1,026.

• 1920x1080: Claude 2,691, GPT-5.x 2,448, Gemini 1,548, Qwen3-VL about 2,042.

• 3840x2160 (4K): Claude 4,784 (the cap), GPT-5.x at most 3,000 (the 2,500-patch cap), Gemini 1,548, Qwen3-VL about 2,560 by default or about 8,100 with the high-resolution flag.

Gemini looks odd. By the page's formula, a 512x512 image costs 4 tiles, the same as 1024x1024, and any 16:9 image above 384 px costs 6 tiles. Gemini 3's media_resolution setting may change this, so check with its token counter before you rely on it.

In dollars, on Claude at Sonnet 5.5's $2 per million input tokens, 1,000 images cost about $0.72 at 512x512, $2.74 at 1024x1024, $5.38 at 1080p and $9.57 at 4K. Opus 5.5 at $4 per million doubles those figures, and Haiku 5.5 at $0.10 per million (for prompts up to 100,000 tokens) makes them one twentieth. I did not read price tables for the other providers, so I show tokens only.

One image is cheap. The next section is why that stops being true in a long chat.

Why a chat makes it worse

A chat resends everything. Anthropic's Messages API is stateless, and its vision page says Claude has access to every image from earlier turns. OpenAI's previous_response_id chaining still bills all previous input tokens. So an image you paste at turn 2 is part of the input on every turn after it.

Take the assumed chat from my last article: a 1,000-token system prompt, 200-token messages, 400-token replies, Sonnet 5.5 prices ($2 per million input tokens).

• One 1080p screenshot, 2,691 tokens, pasted at turn 2 of a 20-turn chat, is sent 19 times: 51,129 tokens from a single image.

• Five such screenshots, pasted at turns 2, 4, 6, 8 and 10, take the chat from 138,000 to 339,825 input tokens, about 2.5 times more.

• Uncached input cost goes from $0.276 to $0.680. With a warm cache the same chat goes from $0.044 to $0.097.

A single screenshot is small. Five of them in a chat that keeps going more than double the input.

Anthropic also warns about payload size. If images are base64, the full bytes are resent each turn. The Files API keeps the request small by sending a file_id, but the docs speak about request size and latency, not about token count.

What the docs say about caching images

• Anthropic. Images in user turns are cacheable content. But the invalidation table says adding or removing images anywhere in the prompt affects message blocks. It leaves the tools and system caches intact and invalidates the messages cache. The docs do not spell out whether appending an image in your newest turn counts, so test it: watch cache_read_input_tokens on the turn you add an image. As an illustration of the downside, if each of my five image turns rewrote the whole history at the 1.25 times write price, the cached cost of that chat would be $0.204 instead of $0.097.

• OpenAI. The cached context includes conversation history with text, images, documents and audio. Images are not listed as a cache breaker, but the general rule applies: the entire rendered prefix must match.

• Google Gemini. Implicit caching is on by default for Gemini 2.5 and newer, with a minimum of 4,096 tokens on the Gemini 3 models listed and 2,048 on 2.5. The page does not say how media is treated.

• DeepSeek. Caching is automatic and prefix-based. On deepseek-flash off-peak, a cache hit costs $0.003 per million tokens against $0.15 for a miss. The pages I read are silent on images.

• Qwen. The vision page says nothing about caching images.

Gotchas that surprised me

1. OpenAI low is not always cheaper. On the gpt-5.4 family, the docs say low uses a 2,048 px, 6,144-patch budget, so it can use more tokens than high.

2. Anthropic's many-image limit includes old turns. Above 20 images in one request, a stricter per-image size limit applies, and the docs say images from earlier turns that you resend count, as do screenshots inside tool results. Oversized ones fail with an error that mentions many-image requests. Staying under 2,000 px per side avoids it.

3. High resolution is on by default. On the newest Claude models you get the larger tier automatically, so the same screenshot can cost far more than on an older model.

4. Image tokens count toward rate limits. OpenAI says so explicitly, which matters if you send many screenshots.

Five ways to cut it

1. Resize before you send. On Claude, a 1920x1080 screenshot resized to 1456x819 costs 1,560 tokens instead of 2,691, 42% fewer. In my five-screenshot chat that cuts input from 339,825 to 255,000 tokens and the warm-cache cost from $0.097 to $0.074. Check that small text stays readable, since Anthropic notes that resizing can hurt legibility.

2. Crop to what matters. One tight crop of the error message beats a full-screen capture.

3. Use each provider's resolution control. OpenAI detail (mind the low surprise above), Gemini 3 media_resolution, Qwen max_pixels and the high-resolution flag.

4. Decide early what stays in the chat. This one is my advice, not a documented feature. Once the model has extracted what it needs from a screenshot, start a fresh chat and paste its text summary instead of carrying five images through every turn. Removing images from the middle of an Anthropic chat may invalidate the cache once, so do it at a natural break.

5. Measure per image. Send the same prompt with and without the image and compare input_tokens in the usage fields. Then watch your cache read tokens on the turns where you add or remove images.

What I could not verify

I ran no live token counts. The counts for each provider come from that provider's documented formula, and the OpenAI, Qwen and Gemini numbers for 1080p are my arithmetic from those formulas. The cost examples use assumed message sizes and one 1080p screenshot size. I did not verify whether appending an image to your newest turn breaks Anthropic's cache. I found no image token formula for DeepSeek, no per-setting counts for Gemini's media_resolution, and nothing on image caching for Gemini, Qwen or DeepSeek. I did not read documentation from xAI, Mistral or Meta, or OpenAI's price tables. Prices and limits change.

Sources, all accessed October 13, 2026: Anthropic: Vision; Prompt caching; Pricing (platform.claude.com/docs). OpenAI: Images and vision; Prompt caching; Conversation state (developers.openai.com). Google: Image understanding; Context caching (ai.google.dev). Alibaba Cloud Model Studio: Image and video understanding. DeepSeek: Models and pricing; Context caching (api-docs.deepseek.com).

How many screenshots does your longest chat carry, and do you know what they cost on every turn?

#AI #LLM #AIEngineering #Multimodal

Work with me: I take 1:1 calls on mobile architecture reviews, mobile and AI questions, and startup tech consults. Book a slot on Topmate: topmate.io/sairamkotha

This article follows that cost from the API docs to the bill: why each turn costs more than the last, what prompt caching fixes, what it does not, and five ways to cut the cost.

How to read it: provider behavior comes from Anthropic's and OpenAI's documentation and pricing pages, which I read on October 12, 2026. The cost examples are my arithmetic with assumed message sizes, not measurements. I ran no live test. Where a page was silent, I say so.

Every turn resends everything

Anthropic's docs say the Messages API is "stateless": you always send the full conversation history. Each turn's input is "all previous conversation history plus the current user message," and the reply becomes part of the next turn's input.

OpenAI's API behaves the same way. Requests are independent unless you pass earlier output back in. Its previous_response_id option chains responses for you, but the docs say "all previous input tokens for responses in the chain are billed as input tokens." Chaining saves you the plumbing, not the tokens.

Chat apps may hide this. Anthropic notes that interfaces like claude.ai can manage the window on a rolling first-in, first-out basis. This article is about the API, where you see and pay for every token.

The per-turn math

Take a modest chat, with assumed sizes: a 1,000-token system prompt, 200 tokens per user message, 400 tokens per reply. Use Sonnet 5.5's published prices: $2 per million input tokens and $10 per million output tokens.

• Turn 1 sends 1,200 input tokens. Input cost: $0.0024.

• Turn 10 sends 6,600. Turn 50 sends 30,600. Turn 100 sends 60,600, which is $0.12 for input alone.

• The reply costs the same $0.004 at every turn. Input overtakes output once the prompt passes 2,000 tokens, about turn 3.

Now add it up. Over 100 turns the conversation holds about 61,000 unique tokens, but you send about 3,090,000 input tokens, roughly 51 times more. Five times the turns (20 to 100) means about 22 times the input tokens. The total, without caching: $6.58, of which $6.18 is input.

The growth is quadratic. A chat that is twice as long costs about four times as much on input.

What prompt caching changes

Caching lets the provider reuse the processed prefix of your prompt instead of recomputing it, at a lower price.

On Anthropic, a 5-minute cache write costs 1.25 times the base input price, and a cache read costs 0.1 times, or 0.05 times on Opus 5.5 and Sonnet 5.5. Anthropic's pricing page says caching "pays off after one cache read" for the 5-minute duration. The cache is refreshed at no extra cost each time it is used. For a conversation, you add one top-level cache_control field and the breakpoint moves forward as the conversation grows.

OpenAI turns caching on by default for supported models. Its docs say cached input tokens are discounted up to 95%, and that on GPT-5.6 and later, reads cost 0.1 times and writes 1.25 times the uncached rate.

Same 100-turn chat, Sonnet 5.5, with the cache staying warm: $0.85 instead of $6.58. Input cost drops about 13.6 times. At turn 100, the input bill is $0.0075 instead of $0.1212.

Where caching does not help

This is the part that surprises people.

1. Output is never discounted. In the example, $0.40 of the $0.85 is output.

2. A pause can cost more than no caching. Anthropic's default cache lives 5 minutes, refreshed on use. If you come back later, the next turn rewrites the whole history at the write price. At turn 100, that is $0.1515 against $0.1212 uncached, about 25% more. A 1-hour write costs 2 times the base price, \(0.2424. If the cache expired on every 10th turn, my 100-turn total rises from \)0.85 to $1.65.

3. Edits break the prefix. Anthropic requires "100% identical prompt segments." Changing tool definitions, the system prompt, images or thinking settings invalidates the cache from that level on. OpenAI says to append new messages rather than rewriting earlier turns. Editing an old message in the middle of a chat throws away everything after it.

4. It does not shrink the window. The context docs say it directly: cached prefixes "still occupy the context window," and caching "changes what you pay for those tokens, not whether they count."

5. It does not fix quality. The same page says that as token count grows, "accuracy and recall degrade, a phenomenon known as context rot." A cheaper long chat is still a long chat.

6. Short prompts are not cached, silently. Below 512 tokens on Sonnet 5.5, Opus 5.5 and Haiku 5.5 there is no error. Check that cache_creation_input_tokens or cache_read_input_tokens is above zero.

7. Haiku 5.5 is priced by prompt length. A request over 100,000 input tokens pays higher prices, and the docs say cache reads and writes count toward that length.

Five ways to cut it

1. Turn caching on and only append. Put stable content first and dynamic content, like timestamps, last. Never rewrite earlier turns. On Anthropic, a mid-conversation system message is appended to the end, so it "does not invalidate any cached prefix that came before it." Use that for instructions that arrive later, instead of editing the system prompt.

2. Trim what you resend, including the model's own replies. The system prompt, tool definitions and old tool results ride along on every request. Cut tools you rarely use or defer them (see my earlier article on the tool-definition tax). Anthropic's context editing can clear old tool results and old thinking blocks.

The model's replies ride along too. In my example they are two thirds of each turn's growth, and output costs 5 times input per token on Sonnet 5.5 with no caching discount. Halve the reply from 400 to 200 tokens and the 100-turn total falls from $6.58 to $4.40 uncached, and from $0.85 to $0.51 cached. Once caching is on, replies are a bigger share of what you still pay.

Ways to get shorter replies: tell the model in the system prompt to answer briefly, use a lower effort setting where the model has one (see my reasoning-effort article), or load a terse-style instruction pack. Some Claude Code users run community skills such as caveman or ponytail for this. I have not measured their savings, so compare output_tokens before and after on your own tasks, and keep terse mode for routine turns because it can drop detail you need. max_tokens is not a substitute: it stops generation at the limit, it does not make the answer shorter.

3. Compact before the window fills. Anthropic calls server-side compaction the primary strategy for long conversations. It summarizes earlier turns on the server so the conversation can continue. It is in beta for Claude 4.6 and later. I did not verify how compaction interacts with an existing cache.

4. Start a fresh chat for each task. This one is my advice, not a documented feature. In my example, one 100-turn chat sends 3.09 million input tokens. Ten separate 10-turn chats send 390,000, about 8 times fewer, with input cost of $0.78 instead of $6.18. The trade-off is that you carry over only what you paste in, so write a short summary of what matters.

5. Measure it. Log input_tokens, cache_read_input_tokens and cache_creation_input_tokens on every turn. Total input is their sum. Anthropic's token counting API estimates a request before you send it. A cache read rate near zero on a long chat means you are paying full price for history.

What I could not verify

I ran no live test and counted no real conversation. The cost numbers use assumed message sizes, a cache that never expires except where I say so, and published prices that change. I did not read how consumer chat apps count usage against subscription limits. I did not verify how compaction interacts with caching. I did not measure the savings of any terse-style skill or prompt. I did not read Gemini, DeepSeek or Qwen documentation, and I read only the caching and state pages for OpenAI, not its full pricing tables.

Sources, all accessed October 12, 2026: Anthropic: Using the Messages API; Context windows; Prompt caching; Pricing (platform.claude.com/docs). OpenAI: Conversation state; Prompt caching (developers.openai.com).

At what turn count do you start a new chat, and do you know what that decision saves?

#AI #LLM #AIEngineering #PromptCaching

Work with me: I take 1:1 calls on mobile architecture reviews, mobile and AI questions, and startup tech consults. Book a slot on Topmate: topmate.io/sairamkotha

This article follows that text from the MCP spec to the invoice, with numbers from the providers' own documentation and one measurement I ran myself.

How to read it: provider behavior comes from Anthropic's and OpenAI's documentation and the MCP specification, which I read on October 10, 2026. The numbers about Anthropic's tool-search results are Anthropic's own, from its engineering blog and docs. The cost example is my arithmetic with assumed sizes. I ran no accuracy benchmark. Where a page was silent, I say so.

What the model actually receives

In MCP, a server lists its tools with a tools/list request. Each tool carries a name, an optional title, a description, and an inputSchema (JSON Schema), plus an optional outputSchema and annotations. The spec says tools are "model-controlled": the model discovers and calls them from context.

Your client then puts those definitions into the model's prompt. Both big providers say what that costs:

• Anthropic: the price depends on "the total number of input tokens sent to the model (including in the tools parameter)". The extra tokens come from tool names, descriptions and schemas, and the API adds a hidden tool-use system prompt of 286 tokens on Sonnet 5.5, Opus 5.5 and Haiku 5.5.

• OpenAI: function definitions "count against the model's context limit and are billed as input tokens."

So a tool is not free until it is called. It costs its definition, every request.

How big is the tax?

Anthropic's engineering post on advanced tool use gives one example setup of five MCP servers:

• GitHub: 35 tools, about 26K tokens

• Slack: 11 tools, about 21K tokens

• Sentry, Grafana, Splunk: about 3K, 3K and 2K

• Total: 58 tools, about 55K tokens before the conversation starts

Anthropic also says it saw definitions reach 134K tokens internally before optimizing. Sizes vary a lot by server, so treat 55K as one documented example, not a constant. Other sources I found give different numbers for the same GitHub server, which is a reason to measure your own.

I measured one real server. I started @modelcontextprotocol/server-filesystem, sent tools/list, and serialized the result as compact JSON in the shape a provider receives (name, description, input schema):

• 14 tools

• 8,001 characters

• 4,108 of those characters (51%) are descriptions

I did not convert this to tokens. I had no API access in this session to call a token counter, and characters are not tokens. The point is the shape: roughly half of a small server's footprint is prose the model reads on every turn.

Why it is also an accuracy problem

Anthropic's tool-search docs say Claude's ability to pick the right tool "degrades once you exceed 30–50 available tools." OpenAI's function-calling guide suggests "fewer than 20 functions available at the start of a turn," and calls it a soft suggestion.

Anthropic's blog reports its own evals with tool search enabled: Opus 4 went from 49% to 74% and Opus 4.5 from 79.5% to 88.1% on an MCP evaluation. These are vendor numbers on older models, on Anthropic's own test, so read them as direction, not as a promise for your stack.

Price per task, not per request

Worked example. This is my arithmetic with assumed sizes, using Sonnet 5.5's published prices: $2 per million input tokens, $2.50 to write a 5-minute cache entry, and $0.10 to read one.

Assume an agent task of 20 model calls, with 55K tokens of tool definitions in every call (1.1M tokens in total):

• No caching: 1.1M tokens at $2 per million is about $2.20 for definitions alone.

• Cached, with the definitions as a stable prefix: one write (55K at $2.50 per million) is $0.14, plus 19 reads (1.05M at $0.10 per million) is $0.10. About $0.24.

• Tool search, uncached: Anthropic says it typically cuts definition tokens by over 85 percent. Taking 85%, 20 calls carry about 165K tokens, about $0.33.

Two things stand out. Caching the definitions and deferring them land in the same range, roughly 7 to 9 times cheaper than the naive case. And the cached prefix only pays while the cache lives: the 5-minute entry has to be refreshed by traffic.

Caching does not fix accuracy. A cached 58-tool menu is still a 58-tool menu for the model to choose from.

The hidden interaction with the cache

Anthropic's caching docs list what invalidates the cache. The prefix runs tools, then system, then messages, so modifying tool definitions invalidates the entire cache. Changing tool_choice or disable_parallel_tool_use invalidates the messages cache.

This matters for MCP because the tool list is not static by contract:

• A server may send a list_changed notification. A client that re-fetches and re-sends a different list changes the prefix.

• The MCP spec asks servers to "return tools in a deterministic order" because it "improves LLM prompt cache hit rates." A server that returns the same tools in a shifting order can quietly break your caching.

• Anthropic notes that for an mcp_toolset you do not control the order of tools within it, so the breakpoint goes on the toolset entry.

Fixes the docs actually offer

1. Load fewer tools. Anthropic says to consolidate related operations (one tool with an action parameter instead of three) and to prefix names by service so groups stay clear.

2. Defer what you rarely use. On Anthropic, set defer_loading: true and add a tool-search tool. Deferred definitions stay out of the cached prefix and are appended when discovered, so caching is preserved. Documented limits: up to 10,000 deferred tools, five matches returned by default, and a deferred tool cannot also carry cache_control (that returns a 400). Keep your 3 to 5 most-used tools loaded up front.

3. OpenAI has a counterpart: tool_search with defer_loading and namespaces. The page says only gpt-5.4 and later support it, and it does not say how it interacts with caching.

4. Restrict instead of rebuild. OpenAI's allowed_tools limits what is callable without changing the tool list you send, which helps prompt caching.

5. Write shorter descriptions, but not carelessly. Anthropic's guidance is the opposite of what you would guess: it calls detailed descriptions "by far the most important factor" in tool performance. Trim by dropping tools, not by starving the descriptions of the tools you keep.

When not to bother: Anthropic says standard tool calling is a better fit with fewer than 10 tools, or when definitions are small.

A checklist you can run this week

1. Count your tools across all connected servers, and write the number down.

2. Read usage.input_tokens on a request that does nothing useful, such as "hi", with tools attached. That is your tax per call.

3. Multiply by calls per task, then by price, for your real cost per task.

4. Pin tool order and check cache_read_input_tokens stays high across turns.

5. If you pass 10 tools or 10K tokens of definitions, try deferred loading and compare cost and correct-tool rate on 20 real tasks.

6. Re-measure after every MCP server upgrade. Descriptions change between versions.

What I could not verify

I ran no accuracy benchmark and no end-to-end token count. My only measurement is the character size of one server's tool list. The 55K and 85% figures and the eval gains are Anthropic's, from its own post and docs. I did not read DeepSeek's or Alibaba's tool pages, so I make no claim about them. Third-party posts quote other token counts for the same servers, and I did not use them. Features like tool search, supported models and prices change, so recheck them before relying on any number.

Sources: Model Context Protocol specification, Tools (modelcontextprotocol.io). Anthropic: Tool use overview, Tool search tool, Tool use with prompt caching, Define tools, Pricing (platform.claude.com/docs); Advanced tool use (anthropic.com/engineering). OpenAI: Function calling (developers.openai.com). All accessed October 10, 2026.

How many tools does your agent load by default, and do you know what they cost per request?

#AI #MCP #LLM #AIEngineering

Work with me: I take 1:1 calls on mobile architecture reviews, mobile and AI questions, and startup tech consults. Book a slot on Topmate: topmate.io/sairamkotha

This article compares the documented controls at Anthropic, OpenAI, DeepSeek and Alibaba Qwen, and shows why the per-token price is not the cost of a task.

How to read it: every behavior below comes from each provider's own documentation, which I read on October 9, 2026. The cost example is my own arithmetic with assumed sizes, not a measurement. I ran no benchmark and compared no answer quality. Where a provider's page was silent, I say so instead of filling the gap.

The one fact most of them share

Reasoning tokens are billed as output tokens. That is the expensive side of the price list. Three of the four providers say it outright:

• Anthropic: the tokens Claude spends reasoning are billed as output tokens, even when the thinking text is not returned.

• OpenAI: reasoning tokens are not visible through the API, but they occupy the context window and are billed as output tokens.

• Alibaba Model Studio: thinking content is billed as output tokens.

DeepSeek's thinking page does not say how reasoning is billed, so I make no claim for it.

So a setting that makes the model think more is a setting that raises your output bill, whatever the per-token price says.

The four controls side by side

The values in a row are not interchangeable across rows. "High" at one provider is not "high" at another.

What each provider actually documents

Anthropic: soft guidance, not a budget

Anthropic calls effort "a behavioral signal, not a strict token budget." Even at low effort Claude still thinks on hard problems, only less. Effort also covers more than thinking: it changes text, tool calls and arguments. Lower effort means fewer and terser tool calls.

Two controls bound cost, per the docs. max_tokens is a hard cap on thinking plus response text combined. Effort is soft guidance on how much of that to spend on thinking. The docs say you do not set a thinking token budget.

Other documented details worth knowing:

• Thinking counts toward max_tokens. If you see stop_reason: "max_tokens", raise the cap or lower the effort.

• The billed thinking tokens are the full thoughts, not the summary you see. You can read the count in usage.output_tokens_details.thinking_tokens.

• Levels are recalibrated per model. For Sonnet 5.5 the docs say a level does not produce the same amount of thinking as the same level on Sonnet 5, and to rerun your own sweep.

• Turning off up-front thinking works differently per model. On Sonnet 5.5 you send thinking: {"type": "between_tools"}, and at xhigh or max that returns a 400.

OpenAI: model-dependent values, and a limit that can eat the answer

OpenAI's reasoning.effort values depend on the model. GPT-6 Astra does not support none; setting it returns HTTP 400. GPT-6.1 Sol supports neither none nor minimal. Reasoning tokens show up under usage.output_tokens_details.reasoning_tokens.

The notable behavior is max_output_tokens. It caps reasoning plus visible output. If the model hits it, the response comes back incomplete, and that can happen before any visible text exists. In OpenAI's words, you may pay for input and reasoning tokens without getting a visible answer. The docs suggest reserving at least 25,000 tokens for reasoning and output when you start experimenting.

DeepSeek: effort is a three-step ladder in disguise

DeepSeek's thinking mode is on by default with effort high. The documented mapping is the interesting part: minimal and low map to low; medium, high and xhigh map to high; max maps to max. So seven names collapse to three real settings.

In thinking mode, temperature and the penalty parameters are accepted but ignored. The chain of thought comes back in reasoning_content. If your request includes tools, you must pass reasoning_content back from all previous turns, or the API returns a 400. The page says nothing about reasoning-token billing or a thinking token budget.

Alibaba Qwen: the only documented thinking cap

Model Studio's thinking_budget sets the maximum number of tokens for the thinking process. When the limit is reached, the model outputs its response. The console shows a range of 1 to 32,768. The API docs say the default is the model's maximum thought-chain length. It is documented for the Qwen3 family and several other series, and enable_thinking toggles hybrid models; thinking-only models cannot be switched off.

Among the four providers, this is the only one I found that documents a cap aimed at thinking alone. Anthropic and OpenAI cap total output; Qwen caps the thinking part.

Why per-token price is not task cost

Worked example. This is my arithmetic with assumed sizes, using Sonnet 5.5's published output price of $10 per million tokens.

Assume a task that produces a 500-token visible answer.

• Little thinking, 500 thinking tokens: 1,000 output tokens, about $0.010.

• Heavy thinking, 8,000 thinking tokens: 8,500 output tokens, about $0.085.

Same model, same price list, 8.5 times the cost for that request. Over 1,000 such requests that is $10 versus $85. The token counts are my assumptions, not measurements, and real thinking amounts vary by task.

The hidden effort cost: it can break your cache

Anthropic documents a side effect most people do not expect. The resolved effort value is rendered into the prompt, so changing effort between requests invalidates the cache. The docs include an example run: with a 3,546-token cached passage, the second request read all 3,546 tokens from cache, and after switching effort from medium to low, the third request wrote 3,546 tokens again and read none.

At Anthropic's published Sonnet 5.5 cache prices, a write costs $2.50 per million and a read $0.10 per million, 25 times more per token. That is my arithmetic on the ratio, not a quoted figure.

The documented advice: hold top-level effort constant within a cached conversation, and vary it across workloads instead. On supported models, a per-message effort change, in beta, keeps the cache. It needs the beta header mid-conversation-output-config-2026-07-01. I did not find an equivalent cache statement in the other three providers' pages I read, so I make no claim about them.

A checklist you can run this week

1. Read the reasoning count in the usage object on every request, and log it next to the answer length.

2. Set the effort explicitly. Do not rely on defaults, which differ by provider and even by model within one provider.

3. Run the same 20 real tasks at two settings and compare cost per correct answer, not cost per request.

4. Size the output cap for thinking plus answer. A cap that fits a no-thinking reply can cut off a thinking one.

5. Keep effort constant inside a conversation that depends on caching.

6. Rerun the sweep when you change models. Anthropic says so for Sonnet 5.5; assume the same elsewhere.

What I could not verify

I ran no requests and measured no quality or cost differences between effort levels. I did not read pricing pages for OpenAI, DeepSeek or Alibaba, so the only prices used are Anthropic's. Alibaba's English help page gave the thinking_budget range; a second Alibaba page returned a 403 error and I did not use it. Claims I saw in blogs, such as Artificial Analysis cost-per-task comparisons are not in this article because I did not check them against a primary source. Beta features, default levels and limits change, so recheck before relying on any value.

Sources: Anthropic API documentation: Effort, Thinking, Steering thinking (platform.claude.com/docs). OpenAI API documentation: Reasoning models (developers.openai.com). DeepSeek API documentation: Thinking Mode (api-docs.deepseek.com). Alibaba Cloud Model Studio: Deep thinking (alibabacloud.com/help). All accessed October 9, 2026.

When you pick an effort level, do you sweep it on your own tasks or trust the default?

#AI #LLM #AIEngineering #PromptEngineering

Work with me: I take 1:1 calls on mobile architecture reviews, mobile and AI questions, and startup tech consults. Book a slot on Topmate: topmate.io/sairamkotha

This article explains how compaction works in Anthropic's documented tools, what it keeps, what it loses, and when it actually saves money. It also shows the costs with simple arithmetic on published prices.

How to read it: every behavior below comes from Anthropic's API and Claude Code documentation, which I read on October 8, 2026. The cost example is my own arithmetic with assumed sizes, not a measurement. Anthropic does not publish how the claude.ai chat app manages long chats, so this covers the API and Claude Code, not the app.

Why anything has to be compacted

Two documented facts create the problem:

• The API is stateless. The client re-sends the history with every request.

• Accuracy and recall degrade as the amount of text grows. Anthropic calls this context rot, and its compaction page says the aim is to keep the active context small because response quality degrades as a conversation grows.

So a long chat is both more expensive per message and, past some point, worse. The usual answers are to start over, to delete old material by rule, or to summarize it. Compaction is the third.

What compaction does

Per Anthropic's compaction overview, the older turns are replaced by a summary that Claude writes on the server, so you need no summarization code of your own.

The old messages are not edited. They are swapped out. After compaction, the model reads the summary where those messages used to be, followed by whatever came after.

Three ways to trigger it (API)

1. On demand. Your application decides when. You send the conversation with a compaction parameter set to summarize. Claude does not reply. It returns a single block holding the summary text and a signature, with stop_reason set to "compaction". This is in beta and Anthropic recommends it where available.

2. At a token threshold. You set a trigger and the API compacts by itself inside an ordinary request when input tokens reach it. The default trigger is 150,000 input tokens and the minimum allowed is 50,000. The check runs at the start of each sampling step, so it can happen more than once in one request.

3. Your own summarizer. You write the summarization call, the prompt and the history rewrite yourself.

Both built-in options are marked beta in the docs.

The swap, step by step (on demand)

1. Send the current conversation plus compaction: {"type": "summarize"}.

2. The API summarizes every message in that request once and returns only the summary block.

3. On your next request, put that block first in messages and remove the messages it replaced. Send it back exactly as returned, signature included.

4. The model now sees: summary, then your next turn.

Three details from the docs matter in practice:

• The summarization call is a normal billed request. It uses the model, system prompt and tools of the request, and the docs say to allow several thousand max_tokens because thinking counts toward the cap.

• The swap can fail silently. Anthropic warns that if summarized messages stay after the block, they are sent to Claude again, and if a later request leaves the block out, Claude gets no summary. Neither raises an error.

• A summary is only produced if the call ends normally with text. If it hits max_tokens or tries to call a tool, you get a 200 response with empty content and have to resend.

You can control what the summary keeps

Without instructions, the API uses its own summarization prompt. A custom instructions string, up to 16,384 characters, replaces that prompt entirely. Anthropic's example tells the summarizer to preserve every entity and field name agreed so far and the user's latest open request.

You can also keep the most recent turns word for word. No parameter does it. You pick a cut point, send only the older messages for summarizing, and put the block in front of the turns you kept. The docs note the tradeoff: kept turns go back at full length, so the more you keep, the less room the compaction frees.

What gets lost

The docs are direct about this:

• Images, documents, uploaded container files and fetched URLs inside summarized messages are gone once the block replaces them. Restate or re-upload anything a later turn needs.

• Instructions given only in the conversation are summarized with everything else. If they matter, state them again.

• Anthropic documents a custom prompt for the case where the default summary "drops something a later turn needs." That is a plain admission that the default can drop things.

My inference, not Anthropic's wording: a summary is lossy by nature. Exact numbers, file names and edge-case decisions are the first things to check after a compaction.

Compaction is not the only tool

Context editing is the rule-based alternative. Instead of summarizing, it clears old content. Its tool-result clearing strategy starts at 100,000 input tokens by default, keeps the last 3 tool uses, and replaces cleared results with placeholder text. Anthropic's page says server-side compaction is the primary strategy and context editing is for finer control. The two cannot be combined on one request. Clearing tool results also invalidates the prompt cache, so the docs advise setting a minimum amount to clear.

In Claude Code: /compact

Claude Code's documented command is /compact [instructions]. Examples in the docs: /compact Focus on code samples and API usage, or a "Compact instructions" section in CLAUDE.md that says what to focus on.

What survives, per the docs:

• Kept or re-loaded from disk: the system prompt and output style, project CLAUDE.md, auto memory, and the plan from plan mode.

• Re-read after compaction: up to five recently read or edited files, most recently modified first. A file over 5,000 tokens comes back as a path reference, not its content.

• Skills re-injected: capped at 5,000 tokens per skill and 25,000 in total, oldest dropped first.

• Summarized away with everything else: path-scoped rules, nested CLAUDE.md files and hook-added context.

Practical consequence, from the docs: if a rule must persist across compaction, put it in the project-root CLAUDE.md.

What compaction costs

Claude Code sends the summary request with the same system prompt, tools and history as your conversation, plus a summarization instruction. So the request is as big as your context. Anthropic's caching page explains the effect:

• While the cache is warm, the request reads the prefix from the cache, so it "costs a fraction of what the context size suggests" and mostly spends time generating the summary.

• After a break longer than the cache lifetime, there is no cache, so the whole history is reprocessed as uncached input. The docs say this is why /compact costs the most when you resume an old session.

• After compaction, the next turn rebuilds the cache for only the short summary.

The docs also say compaction by design invalidates the conversation part of the cache, because the new history does not share a prefix with the old one.

Worked example (my arithmetic, assumed sizes)

Assume Sonnet 5.5 prices from Anthropic's pricing page: $2 per million input, $10 per million output, $0.10 cache read, $2.50 cache write. Assume a 150,000-token history and a 3,000-token summary. The 3,000 is my assumption, not a measured size.

• Compaction request, warm cache: 150,000 cache-read tokens = \(0.015, plus 3,000 output tokens = \)0.03. About $0.045.

• Compaction request, cold cache: 150,000 uncached input = $0.30, plus $0.03 output. About $0.33. This ignores the cache-write surcharge, so it is a floor.

• Rebuilding the cache for the 3,000-token summary: about $0.0075.

• Each later turn re-reads either 150,000 tokens ($0.015) or 3,000 tokens ($0.0003). The saving is about $0.0147 per turn.

On those assumptions a warm compaction pays for itself in about four turns. A cold one needs roughly 23. Real results depend on summary size, how many turns follow and cache timing. This is an illustration, not a benchmark. It also ignores the quality benefit, which cost arithmetic cannot show.

Rules I would follow

1. Compact at a natural break, between tasks, not in the middle of one. Anthropic's own caching page recommends exactly this.

2. Compact while the cache is warm. Cold compaction of a huge context is the expensive case.

3. Tell it what to keep. Pass focus instructions, or write them in CLAUDE.md.

4. Put rules that must last in project-level files, not only in chat.

5. Check the summary for exact values before trusting it.

6. Restate images, uploads and fetched pages the next step needs.

7. If you only want a fresh start, /clear costs nothing, and /rewind returns to an earlier cached point without a new summary.

What I could not verify

I ran no compaction and measured no quality loss or cost. I did not verify how the claude.ai app handles long chats. The docs say only that chat interfaces can manage context on a rolling first-in, first-out basis, and that is not the same as summarizing. The beta features, header names and thresholds can change. The worked example uses assumed sizes. I did not read Claude Code's source, so I make no claim about the exact summarization prompt it uses.

Sources: Anthropic API documentation: Compaction overview, Compaction on demand, Compaction at a token threshold, Compaction that keeps recent turns, Context editing (platform.claude.com/docs). Claude Code documentation: Commands, Explore the context window, How Claude Code uses prompt caching, Manage costs (code.claude.com/docs). Pricing from Anthropic's pricing page. All accessed October 8, 2026.

When a chat of yours gets long, do you summarize, clear or push through?

#AI #LLM #ClaudeCode #AIEngineering

Work with me: I take 1:1 calls on mobile architecture reviews, mobile and AI questions, and startup tech consults. Book a slot on Topmate: topmate.io/sairamkotha

This article does that arithmetic with Anthropic's published prices, shows where the savings are real, and shows the one condition that decides whether they survive contact with production.

How to read it: prices, benchmark scores and customer quotes come from Anthropic's own pages and are the company's claims. The worked examples are my arithmetic on those prices, not measurements. I tested nothing.

What Anthropic launched

Documented on Anthropic's Haiku 5.5 page and pricing page:

• Haiku 5.5 costs $0.10 input and $0.50 output per million tokens for prompts up to 100,000 tokens. Above 100,000 tokens, it costs $0.50 input and $2.50 output.

• Anthropic says it costs around 75% less to run than Haiku 4.5 on average, and is priced 90% lower for requests up to 100,000 tokens and 50% lower above that.

• It is the first Haiku-class model with adjustable effort controls, and Anthropic calls it its fastest model at standard speed.

• Anthropic describes it as designed for high-volume, cost-sensitive tasks such as summaries, compaction, database queries and classification, and says it "pairs well with Opus 5.5 and Sonnet 5.5 as a subagent on coding work."

• The same page says Sonnet 5.5 and Opus 5.5 "remain better choices for complex agentic coding tasks."

For comparison, the pricing page lists Sonnet 5.5 at $2 input and $10 output, and Opus 5.5 at $4 input and $20 output, per million tokens.

One subtask, three prices

Take a simple subtask a lead agent might hand off: read a 50,000-token document and write a 1,000-token summary.

• Opus 5.5: about $0.22.

• Sonnet 5.5: about $0.11.

• Haiku 5.5: about $0.0055.

Same job, 20 times cheaper than Sonnet and 40 times cheaper than Opus. Because the prompt stays under 100,000 tokens, it lands in the cheaper tier. At 150,000 tokens the same job costs about $0.078 on Haiku, which is still cheaper than Sonnet's $0.31 but a much smaller gap, about 4 times. Keeping subagent prompts narrow is part of the saving.

A whole task with a lead and eight subtasks

Now an illustrative task: a lead model plans (10,000 tokens in, 2,000 out), then eight subtasks of the kind above run.

• Sonnet 5.5 as lead and doing every subtask: about $0.92.

• Sonnet 5.5 as lead, Haiku 5.5 on the eight subtasks: about $0.084, roughly 91% lower.

• Opus 5.5 as lead and doing every subtask: about $1.84.

• Opus 5.5 as lead, Haiku 5.5 on the subtasks: about $0.124, roughly 93% lower.

Most of the bill was the repeated reading, so moving that work to the cheap model removes most of the cost. This is arithmetic, not a benchmark. It assumes the subtasks suit Haiku and that every one succeeds.

The catch: cheap only helps if you can tell when it's wrong

A cheap model that is sometimes wrong can still save money, if you can detect the wrong answers and rerun them on the bigger model. The break-even is simple. If Haiku costs a fraction h of the big model's cost for the same subtask, rerunning failures on the big model beats using the big model throughout as long as the failure rate is below 1 minus h. For Sonnet in the example, that is 95%. For Opus, 97.5%.

That sounds generous, and it is, but it hides the condition: the failures must be detected cheaply. A wrong summary that nobody catches does not cost $0.0055. It costs whatever it does downstream. So the real design question is not "which model is cheap" but "what is my cheap check". Examples that work: a test that must pass, a schema the output must validate against, a number that must appear in the source text, a database query that must return rows. Examples that don't: "read it and see if it looks right."

This is the same lesson I wrote about earlier this week for coding agents: trust the checker you did not let the agent write.

What the capability gap looks like

Anthropic's own table shows the trade. On Terminal-Bench 4.0, a command-line task benchmark, Anthropic reports 39.2% for Haiku 5.5 and 70.6% for Sonnet 5.5. That is why Anthropic says the bigger models remain better for complex agentic coding. The cheap model is for narrow, well-specified subtasks, not for the lead role. I read the benchmark table from a flattened page, so check the original before quoting it.

Customer quotes on the page point the same way. Rogo describes a big model building a deck while a Haiku subagent pulls a revenue line out of a 10-K. Cognition says Haiku 5.5 as the sidekick in its Devin Fusion setup holds a FrontierCode score of 66.2 while cutting cost and latency. That score is for the combined system, not for Haiku alone.

Two price changes landed the same week

Anthropic also halved Sonnet 5.5 cache reads, from $0.20 to $0.10 per million tokens, which it says makes Sonnet 5.5 about 20% cheaper on most agentic work. So the lead model got cheaper too. If you quoted old cache prices anywhere, including in my own earlier posts, check them against the pricing page.

One caution on the headline discount: Haiku 5.5 uses an updated tokenizer and, per Anthropic, "slightly more tokens per task" than Haiku 4.5. The exact increase depends on the workload, so a 90% lower price is not a 90% lower bill.

Rules for using a cheap subagent

1. Give it narrow tasks with a clear definition of done.

2. Keep its prompt under 100,000 tokens to stay in the cheaper tier.

3. Make every output checkable by something cheaper than a person.

4. Escalate failures to the bigger model automatically, and count those reruns.

5. Measure cost per accepted task, not cost per token.

6. Keep the planning and hard reasoning on the lead model.

What I could not verify

I tested nothing. All prices are from Anthropic's pages as of October 8, 2026 and may change. Benchmark scores and customer results are the company's own. Anthropic's page does not state tokens-per-task figures or per-task dollar costs, so my worked examples are assumptions, not Anthropic's numbers. The page did not state a context window or consumer-app availability.

Sources: Anthropic, Claude Haiku 5.5 (anthropic.com/claude-haiku-5-5); Anthropic API pricing (platform.claude.com/docs/en/about-claude/pricing). Accessed October 8, 2026.

Where in your agent would a $0.10 model do the reading, and what would check its work?

#AI #AIEngineering #LLM #Agents

Work with me: I take 1:1 calls on mobile architecture reviews, mobile and AI questions, and startup tech consults. Book a slot on Topmate: topmate.io/sairamkotha

This article walks through all three, with the numbers each company published, and then asks what changes for anyone who builds with these systems or reads their benchmarks.

How to read it: every number below comes from the company's own announcement and has not been independently tested. Where I add analysis of my own, I say so. Where something is not stated or not confirmed, I say that too.

1. Unreleased: OpenAI shows results, not the model

OpenAI published a large set of new mathematical results produced by what it calls an internal frontier model. Its own repository describes the release this way:

• 722 manuscripts, organized into 372 families.

• About 4,000 problems were posed to the model.

• The vast majority of results came from the same procedure, using an unreleased internal OpenAI model.

• On average, each result used three hours of ChatGPT Pro thinking compute.

• Checking was a mix of Lean formalizations where available and human review. Not all results have a Lean formalization.

• OpenAI's own caveat: "Some of the unformalized results could have issues."

OpenAI also said it consulted an independent advisory group on mathematics and AI at the Institute for Advanced Study on how to release the results.

My analysis: when the model is not available, the outputs are the only access anyone has. That makes the checking method the real product. A Lean proof is a program that a computer verifies, so it does not depend on trusting the author. Human review is slower and depends on attention. The release is honest about which is which, and that is the part other teams can copy.

2. Gated: Anthropic puts capability behind verified tiers

Anthropic expanded its Cyber Verification Program. Its description: it "makes advanced cyber capabilities and reduced blocking classifiers available to qualifying security professionals." There are three tiers:

• Defense Access, for work like incident response, malware analysis and vulnerability analysis. Open to security teams, researchers and open-source maintainers.

• Red Team Access, which adds authorized penetration testing and red-teaming. Organizations only, not individuals.

• Specialized Access, reserved for a limited set of verified organizations authorized to test systems such as flight operating systems, power grids and telecom networks. Anthropic says it currently reviews every such organization in depth with the US government.

Anthropic says it will verify all applicants and request proof of the required security controls for the tier. All tiers include Claude Opus 5.5, Sonnet 5.5, Mythos 5.1 and new models going forward. Real-time blocks still apply to actions that could cause physical harm or mass disruption, such as deploying ransomware.

The numbers that matter most come from a test Anthropic ran, CyScenarioBench, which measures whether a model can plan and execute multi-stage cyber operations under realistic constraints. With Claude Opus 5.5:

• Without the program, every task was blocked on the first prompt.

• With Defense Access, 46 of 50 trials were blocked.

• With Red Team Access, no blocks occurred and the model completed 34 of 50 tasks, which Anthropic says is effectively equivalent to its 67.6% success rate.

Anthropic also reports that its Project Glasswing partners found at least 129,000 verified software vulnerabilities between April and July 2026, with an additional 5,500 found through open-source scanning between April and October. More than 33,000 were rated critical or high severity.

My analysis: this is the clearest data point of the week. It is the same model on the same task, and the measured result goes from every task blocked to 34 of 50 completed, depending only on access tier. So a benchmark number for a gated model is a number about the model plus its safeguards, not about the model alone.

3. Open: Mistral previews a trillion-parameter model, with weights to follow

Mistral introduced Mistral Large 4, a 1 trillion-parameter, natively multimodal model with 49 billion active parameters. According to Mistral:

• It accepts text and image input, and the public preview is available now through Mistral Studio.

• Mistral says "We will release the weights by the end of the month." The license is not stated in the announcement.

• Preview API pricing is $1.36 per million input tokens and $4.18 per million output tokens.

• It was trained on 3,800 NVIDIA Grace Blackwell GPUs in Mistral's own datacenters in Europe.

On cybersecurity, Mistral reports that it solves 93% of the challenges in Cybench, a set of 40 exercises drawn from security competitions, "one of the highest scores reported for an open-weight model." On the Artificial Analysis Cyber Index, which Mistral describes as an independent evaluation of how well models find and fix security flaws, it says the model ranks among the top five globally. On Lakera's B3 benchmark, which tests resistance to indirect prompt injection, it resists 93.3% of attacks.

On coding agents, Mistral reports 61.7% on DeepSWE v1.1 and 28.3% on Terminal-Bench 4, and a combined Coding Agent Index score of 49.8%, ahead of DeepSeek V4 Pro 0813 and Qwen3.8 Max.

Mistral also makes a safety claim: that despite its cyber scores, the model's average refusal rate on cyber prompts from JailbreakBench, StrongREJECT and AgentHarm is higher than all open-source models.

My analysis, with two caveats: Cybench here is only 40 exercises, so a 93% score is a small sample. And the weights have not shipped yet. Once they do, the gating in section 2 does not apply, because the safeguards are then whatever is in the weights and whatever the host adds. Mistral's refusal claim is a measurement of the model as released. General ML background, not from Mistral: open weights can be fine-tuned by anyone, which can change refusal behavior. Whether that happens with this model is not something I can say.

4. One more pattern: limited rollouts

Users on X have been posting screenshots of a Gemini 4 Argon option in Google's Antigravity model picker, apparently visible to only some users. I could not find a statement from Google confirming this, so I treat it as unconfirmed. It matters here only as a reminder that "available" often means "available to some accounts."

What connects them

All three releases answer "who gets access" differently, and each one decides what you can verify:

• Unreleased: you verify the outputs. OpenAI's Lean proofs and human review are the only handle.

• Gated: you verify the person. Anthropic checks who you are and which security controls you have, and the model behaves differently by tier.

• Open: you verify the weights. Once Mistral ships them, anyone can test, host and change the model.

None of these is clearly better. They trade off in different directions: reproducibility against control, openness against oversight.

What this means if you build with AI or read benchmarks

1. Ask what access a benchmark used. The same model went from every task blocked to 34 of 50 completed depending on tier. A number without its access conditions is half a number.

2. Treat vendor-reported scores as vendor-reported. Each company above chose its own benchmarks, and none compare like with like.

3. Prefer checks a machine can run. OpenAI itself separates Lean-verified results from reviewed ones.

4. Plan for open weights without assuming they match closed models. Mistral's numbers are reported, and the weights are not out yet.

5. Watch what is not stated. Mistral's announcement leaves out the license and the context window, and both matter for deployment.

What I could not verify

I tested none of these models. Every figure is the company's own claim. I did not verify OpenAI's results, Anthropic's test conditions, or Mistral's benchmark methodology, and no independent group has published results on the Mistral weights, since they are not released. I cannot confirm Gemini 4 Argon beyond user posts. I make no claim about how these three approaches compare on safety or capability.

Sources: Anthropic, "Expanding the Cyber Verification Program" (anthropic.com/news/cyber-verification-program); Mistral AI, "Introducing Mistral Large 4" (mistral.ai/news/mistral-large-4); OpenAI math repository (github.com/openai/math) and OpenAI's announcement post on X. All accessed October 7, 2026.

Which of the three access models would you trust most for your own work, and why?

#AI #AIEngineering #AISafety #OpenSource

Work with me: I take 1:1 calls on mobile architecture reviews, mobile and AI questions, and startup tech consults. Book a slot on Topmate: topmate.io/sairamkotha

You type a question, press Send, and words start appearing within a second. It feels like talking to something that remembers you. It is closer to a fast, stateless function call.

This article follows one message through the whole system in seven levels. Each level adds one layer and assumes only the one before. Stop wherever you have what you need.

How to read it: facts marked Documented come from Anthropic's public API documentation. Sections marked General ML explain standard mechanics the docs do not describe, so treat them as background, not as claims about Claude's internals. Anthropic does not publish how the claude.ai app is built, so this covers the public API path. I measured no benchmarks.

Level 1: The big picture

Pressing Send runs one request. The client sends the whole conversation to a server. The model produces the next message. The text comes back in small pieces while it is still being written.

Three ideas explain most of what you will see:

• Stateless. The server does not keep your chat for you. The client re-sends it.

• Everything is tokens. Limits, speed and price are all counted in them.

• Fixed model. The weights do not change while you chat. Only the text the model reads changes.

Level 2: The request

Tokens. The model reads tokens, which are chunks of text: a short word, part of a long one, or punctuation. The API has a token-counting endpoint, so you can measure a request before you send it.

Documented: the Messages API supports "single queries or stateless multi-turn conversations". You pass earlier turns in the messages parameter and the model generates the next message.

So a second-turn request carries the system prompt, your first question, the model's first answer, and your follow-up. Nothing on the server says "this is turn 2". The history is the request.

That is why a long chat costs more per message than a fresh one. The system prompt and tools repeat every turn, and each reply becomes input for the next turn.

One footnote from the docs: chat interfaces such as claude.ai can also manage the context window on a rolling first-in, first-out basis. That is the only statement the docs make about how the app differs from the raw API.

Level 3: The context window

Documented: the context window is all the text the model can reference when generating a response, including the response itself. It is working memory, separate from the data the model was trained on.

What counts toward it: the system prompt, every message (including tool results, images and documents), your tool definitions, and the reply being generated, including extended thinking. Cached tokens still count. Caching changes the price, not the count.

Sizes: many current models, including Claude Sonnet 5.5 and Opus 5.5, have a 1M-token window, and one request can generate up to 128k output tokens. For those models 1M is the default, billed at standard pricing. Claude Sonnet 4.5 has 200k.

More is not automatically better. The docs say accuracy and recall degrade as token count grows, and call it context rot.

At the limit:

• If the input alone exceeds the window, the API returns a 400 error ("prompt is too long").

• On Claude 4.5 models and newer, if input plus max_tokens exceeds the window, the request is accepted. If generation hits the limit, it stops with stop_reason "model_context_window_exceeded".

• For long agent runs, the docs point to server-side compaction (earlier turns summarized on the server, in beta) and to context editing, which clears old tool results.

Level 4: Inside generation (General ML)

Anthropic does not publish Claude's architecture. This level explains how transformer language models are served in general. The numbers use an invented configuration to show the arithmetic.

Next-token prediction. A language model maps a sequence of tokens to a probability distribution over the next token. One token is chosen, appended, and the process repeats until a stop condition. The API shows this directly: max_tokens is "the maximum number of tokens to generate before stopping", and the stream delivers text a few tokens at a time.

Prefill and decode. Prefill processes your whole prompt. The tokens are known in advance, so the work can run in parallel. Decode generates the reply, and each new token depends on the one before, so it runs sequentially. That is why a long prompt and a long answer cost differently, and why output tokens are priced higher. On Sonnet 5.5 the listed prices are $2 and $10 per million.

The KV cache. In each attention layer, every token produces a key vector and a value vector. To generate the next token, the model attends over the keys and values of all earlier tokens. Recomputing them every step would repeat work, so servers keep them in memory. The size per token is:

2 (K and V) x layers x KV heads x head dimension x bytes per value

Hypothetical example, not Claude's configuration: 32 layers, 8 KV heads, head dimension 128, 2 bytes per value. That is 131,072 bytes (128 KiB) per token. At 100,000 tokens, one conversation holds about 12.2 GiB. At 1,000,000 tokens, about 122 GiB. Accelerator memory is finite and shared across many requests, so long contexts are expensive to hold.

Sampling. Documented: models released after Claude Opus 4.6 do not support setting temperature. A value of 1.0 is accepted for backwards compatibility and other values are rejected.

Level 5: Caching and the cost of a conversation

Documented:

• The cached prefix follows a fixed order: tools, then system, then messages.

• Default lifetime is 5 minutes, refreshed on every reuse. A 1-hour option costs more.

• 5-minute cache writes cost 1.25x the base input price. Reads cost a fraction, 0.1x on most models.

• Prompts below a minimum length are not cached and no error is returned. The minimum is 512 tokens on the newest models, up to 4,096 on others.

• The response reports cache_creation_input_tokens, cache_read_input_tokens and input_tokens. Total input is the sum of the three.

The docs do not describe the implementation. The standard technique in serving stacks is to store the KV tensors for a prompt prefix and reuse them when a later request starts with the same tokens. That matches the documented rule that only an identical prefix hits. One changed token early in the prompt invalidates everything after it.

Worked example, arithmetic on published Sonnet 5.5 prices ($2 input, $10 output, $0.10 cache read, $2.50 cache write per million tokens). A 20-turn agent loop with an 8,000-token system prompt plus tools, 1,500 new tokens per turn, 500 output tokens per turn, and every turn inside the 5-minute window:

• The loop sends 475,000 input tokens across 20 turns, though the final conversation holds only 38,000.

• Without caching, the run costs about $1.05.

• With every turn hitting the cache, about $0.24, roughly 77% less.

• Output is $0.10 of both totals, so as the history grows, input dominates the bill and caching is the lever.

This is an illustration, not a measurement. Real hit rates depend on timing and prefix stability.

Level 6: Thinking, tools and streaming

Thinking. Documented: thinking tokens are a subset of max_tokens, billed as output tokens, and count toward rate limits. With adaptive thinking, the model sets its own allocation, so usage varies by request. Whether earlier thinking blocks stay in later turns depends on the model. When you send back a tool result, you must return the whole unmodified thinking block with it, because blocks are signed and altering one makes the API return an error.

Tools and the agent loop. A model cannot run code or browse. It can only emit a structured request, and something else acts on it.

• Client tools (your own functions, bash, text editor) run in your application. The model replies with stop_reason "tool_use", you run it, and you return a tool_result.

• Server tools (web search, web fetch, code execution) run on Anthropic's infrastructure and return results in the same response.

An agent is a loop: call the API with the full history, get a tool_use, run it, append the tool_result, call again. Every lap re-sends a longer history, which is the arithmetic in Level 5. Tool definitions and tool blocks all count as tokens, and enabling tools adds a built-in system prompt, 286 tokens on Sonnet 5.5 and Opus 5.5 per the pricing table.

Streaming. Set stream to true and the reply returns as server-sent events, in this order: message_start; for each content block, content_block_start, one or more content_block_delta events, then content_block_stop; one or more message_delta events with the stop reason and cumulative token counts; and a final message_stop. Ping events can appear anywhere. The words arrive inside content_block_delta events as text chunks. Tool arguments stream as partial JSON, which the docs say to accumulate and parse once the block stops.

Stop reasons include end_turn (finished naturally), stop_sequence (hit a custom sequence), tool_use (wants a client tool run) and model_context_window_exceeded. The docs describe more, so handle each one your integration can receive.

Level 7: Rules for building on it

1. Own the history. Truncation and summarization are your decisions. The server makes none.

2. Budget the whole window. System prompt, tools, history, tool results and thinking all count. Use token counting before you send.

3. Order the prompt for the cache. Stable content first, volatile last, and no timestamp at the top. Watch cache_read_input_tokens. Zero on a repeated prompt means the cache is not working.

4. Budget output on purpose. Set max_tokens deliberately and handle every stop reason, including a reply cut short.

5. Expect each agent step to cost more than the last, unless caching or context clearing offsets it.

6. Parse streams on block end, and ignore event types you do not recognize.

7. Prune context. Old tool results and stale files add cost and, per the docs, can hurt recall.

What this does not cover

I measured no latency, throughput or accuracy. The cost example is arithmetic on published prices, and the KV example uses an invented configuration. Level 4 is general machine-learning background. I did not verify how claude.ai stores chats, what it adds to the prompt, or how requests are routed inside Anthropic's infrastructure.

Sources: Anthropic API documentation (Streaming messages, Messages, Prompt caching, Context windows, Tool use overview) at platform.claude.com/docs. Pricing from the Sonnet 5.5 launch page on anthropic.com. Prices and limits change, so re-check before relying on them.

Which level did you already know, and where did it get new?

#AIEngineering #LLM #SoftwareArchitecture

Work with me: I take 1:1 calls on mobile architecture reviews, mobile and AI questions, and startup tech consults. Book a slot on Topmate: topmate.io/sairamkotha

What each claim actually says

OpenAI (Tibo, Oct 5, "Day 1/"). "We have optimized the default speed to be ~50% faster across GPT-6 Astra and GPT-6.1 Sol through the subscription across all our products and partners using Sign in With ChatGPT." He adds that no changes are needed and it should be felt within two hours. The post gives no tokens-per-second figure, no benchmark and no cost number, and it describes the subscription default, not API behavior.

Anthropic (Sonnet 5.5 page). Output generation is "30%+ faster than Sonnet 5", and the model "costs up to 30% less per task than its predecessor" for most work. Anthropic attributes this to the model batching tool calls together more, "leading to fewer steps and lower costs". Customer data points on the page: Slack reports about 14% fewer output tokens, Box 12% fewer total tokens, and Balyasny Asset Management measured 121K tokens per answer versus 497K for Sonnet 5. API prices are $2 per million input tokens, $10 output, $0.20 cache reads at the time (cut to $0.10 on October 7-8) and $2.50 cache writes.

"50% faster" is ambiguous

1.5x the throughput means about 33% less time per response. 50% less time means 2x the throughput. The post does not say which one it means, so I would not plan around either reading until the number is defined.

Where the time goes in an agent loop

Wall time for a task is roughly steps x (model time + tool time) per step.

• Faster decoding shrinks only the model-time term.

• Fewer steps, such as batching tool calls, shrinks the multiplier.

• Tool time (tests, builds, network) does not change at all.

Illustrative arithmetic with hypothetical numbers, not a measurement of any model: 20 steps, 10 s of model time and 5 s of tool time per step gives 300 s.

• A 1.5x faster model: 20 x (6.7 + 5) = about 233 s, 22% less.

• 30% fewer steps at the same per-step cost: 14 x 15 = 210 s, 30% less.

• Both: 14 x (6.7 + 5) = about 163 s, 46% less.

A speed-up helps, but cutting steps can help as much, and neither touches tool time.

Cost is a separate axis

Faster serving changes time, not tokens. On a subscription it changes neither price nor tokens. Anthropic's cost claim comes from fewer tokens, steps and tool calls. Balyasny's 497K to 121K is about 4.1x fewer tokens, yet the headline is "up to 30%" per task, so token counts and dollar cost are not the same number. These are vendor-reported figures and I have not reproduced them.

What I could not verify

• Any OpenAI per-token or API pricing for GPT-6 Astra or GPT-6.1 Sol.

• A tokens-per-second figure for either model. Third-party posts quote about 50 tok/s, but I have no primary source.

• Any head-to-head between the two vendors on the same task.

What I would measure instead

Per task: steps, model time versus tool time, tokens per solved task, and pass rate. One data point from my own locked-test bench: Sonnet 5.5 fixed all 6 planted bugs in 3 of 3 runs with 4 to 5 tool calls and about 11 s each. That is n=3 on one small task, so it shows the method, not a ranking.

Sources

• Tibo (OpenAI), Day 1 post: https://x.com/thsottiaux/status/2107158998495748264

• Anthropic, Claude Sonnet 5.5: https://www.anthropic.com/claude-sonnet-5-5

Which of these do you track for your agents today: steps, model time, tokens per solved task?

Work with me: I take 1:1 calls on mobile architecture reviews, mobile and AI questions, and startup tech consults. Book a slot on Topmate: topmate.io/sairamkotha

Two numbers decide what a long-running agent costs and whether it can run at all: how many bytes the KV cache needs per session, and how much of each prompt you pay full price for again. Both moved this year.

1. KV bytes, not the context spec, set the real limit

A 1M-token context window used to be a spec-sheet number. On DeepSeek V3.2 it meant roughly 84 GiB of KV cache per sequence (bf16 estimate). One long agent session could push an entire batch out of HBM.

DeepSeek-V4 needs about 9.6 GiB for the same 1M context, per vLLM's analysis. Hugging Face puts it near 2% of a standard grouped-query-attention cache. DeepSeek-V4-Pro uses about 10% of V3.2's KV memory and 27% of its single-token FLOPs; V4-Flash uses about 7% of the KV cache.

What changed in the design:

• Compressed Sparse Attention: 4x KV compression, with a lightning indexer (running in FP4) picking top-k blocks.

• Heavily Compressed Attention: 128x compression.

• A sliding-window branch keeps the most recent tokens exact.

• FP8 for most KV entries, BF16 only for the RoPE dimensions.

The second-order effect is on serving. An engine now manages several cache types with different compression ratios. vLLM folds five of them into three page-size buckets over 256-token blocks so fragmentation doesn't eat the savings.

2. Most of an agent's prompt doesn't change between turns

A recent arXiv paper on stateful inference (2605.26289) reports that 85 to 95 percent of the prompt is unchanged between turns in agent loops. If that holds for your workload, resending the full history every turn means paying again for tokens you already processed.

Published prompt-caching prices show the size of that gap. For Claude Sonnet 5.5, Anthropic lists $2 per million input tokens and $0.10 per million for cache reads (writes are $2.50). That is after an October 2026 price cut; it was $0.20 when I first wrote this. A cache read now costs a twentieth of fresh input.

Illustrative arithmetic, from the published prices and not a measurement: take a 100K-token prompt where 85% is unchanged and cached.

• Uncached: 100,000 tokens at \(2/MTok = \)0.200

• Cached: 85,000 at \(0.10/MTok (\)0.0085) plus 15,000 at \(2/MTok (\)0.030) = $0.0385

That is about 81% lower input cost on that turn. It ignores the one-time cache write and any output tokens, and your real unchanged fraction will differ.

3. How the two fit together

The first number is about whether a long session fits in memory. The second is about whether you pay for it repeatedly. Both reward the same design habit: keep the stable part of the context stable, and treat cache layout as part of the architecture, not an afterthought.

Anthropic also states that Sonnet 5.5 uses fewer tool calls and costs up to 30% less per task. That is the vendor's claim and I haven't verified it. It is still the right unit: cost per solved task, not price per token.

What I'd do with this

• Model KV bytes per session before comparing parameter counts.

• Measure what fraction of your prompt is unchanged between turns. That fraction is your caching upside.

• Compare models on cost per solved task on your own workload.

Sources

• Hugging Face, DeepSeek-V4: https://huggingface.co/blog/deepseekv4

• vLLM, DeepSeek-V4 serving: https://vllm.ai/blog/2026-04-24-deepseek-v4

• arXiv 2605.26289 (stateful inference)

• Anthropic, Claude Sonnet 5.5 pricing page

For teams running long-context agents: is KV memory or repeated prompt cost your binding constraint today?

Work with me: I take 1:1 calls on mobile architecture reviews, mobile and AI questions, and startup tech consults. Book a slot on Topmate: topmate.io/sairamkotha

The problem with "the agent said it passed"

An agent that edits code and runs tests is grading its own homework. It can fix the bug, or it can loosen the test, skip a case, or report a PASS from a stale run. If you only read the agent's summary, you cannot tell which one happened.

So the rule for this bench is simple: the agent never gets to decide the score. I re-run the tests myself, in a clean copy, against a test file whose hash I recorded before the run.

The setup (about an hour)

1. Plant the bugs. A small inventory module with 6 deliberately planted bugs.

2. Write the tests first. Confirm all 6 fail on the buggy code. If any test passes on buggy code, it isn't testing a bug.

3. Prove the task is solvable. Apply a reference fix in a copy and confirm all 6 pass. Now the task is valid.

4. Lock the tests. Record a SHA-256 of the test file. After every run, re-check the hash. A changed hash means the run is disqualified, whatever the agent reported.

5. Same prompt, fresh copy, every run. No state carried between runs.

6. Record per run: pass/fail from my own re-run, tool calls, wall time.

The harness is plain Python unittest, so there is nothing to install.

The pilot

I ran Claude Sonnet 5.5 three times on this task, as in-session subagents.

3 of 3 fixed all 6 bugs with 4 to 5 tool calls and about 11 seconds each.

What this does not show

• n = 3, one small task, one model. It demonstrates the method. It is not a ranking and says nothing about harder, multi-file work.

• No cost figure. The token totals (about 67K per run) were dominated by fixed session context, with no input/output/cache split, so a dollar estimate would be made up. I'm not publishing one.

• Harness differences matter. Comparing two models through two different agent harnesses compares the stacks, not the models. Any cross-model run needs the same harness.

I also could not run the comparison I wanted: one CLI had an expired login and the other did not support the model on that account. I'd rather say that than quietly swap in a different comparison.

Why the locked-test habit is the useful part

Leaderboards and vendor charts are preference or benchmark numbers on someone else's workload. A bench like this measures what you care about in production: do the tests pass, how many tool calls did it take, how long did it run. Because the tests are locked and you re-run them, the result survives scrutiny.

The takeaway is one line: re-run the tests yourself. The agent's PASS is a claim. Your own run is evidence.

Update, October 9: two reader suggestions for the next version

Two readers found gaps in this pilot. Both are fair, and I'm crediting them by name.

• Hold some tests back (Mike Dabydeen). Visible tests show the agent what a passing answer looks like, so they won't separate models much. Add a second test file the agent never sees: the same bugs, checked with different inputs, such as a negative quantity or an item that isn't stocked yet. Lock it with its own hash. A fix that special-cases the visible inputs passes the first file and fails the second, and the gap between the two scores is the number to compare across models. This pilot did not do that.

• Prove the tests actually ran (Ahmet Özel). A green exit can still come from empty test discovery, or from importing an installed copy instead of the fresh workspace. Record the discovered test IDs, the executed count and the resolved module path next to the test hash, fail the bench on a zero-test run, and keep the six per-test outcomes instead of one Yes per run. The pilot table above shows only Yes, and this pilot did not check these things.

Both go into the next version of the bench.

Next I'll run the same task on other models in the same harness. Which model should go first, and what task would you test?

Work with me: I take 1:1 calls on mobile architecture reviews, mobile and AI questions, and startup tech consults. Book a slot on Topmate: topmate.io/sairamkotha