The Cost of an Agent Step

Agents make many model calls per task, and each call resends a growing context. Write-ups from Oct 19–21 show where the money goes and which levers cut it: prompt-cache hits, a thin deterministic orchestrator that hands narrow work to specialists, smaller post-trained models for those specialists, and serving that keeps a session’s KV cache close. Cloudflare and Shopify also checked that quality held.

Part of This Week in AI & Systems #2: Hardening the Agent Stack.

---

What’s new this week

• Cloudflare (Oct 19) cut the cost of its AI code reviewer by 68%, from $2.43 to $0.77 per review, and the median review time from 4m 09s to 1m 32s.
• Meta (Oct 19) described a “thin orchestrator” for its data-pipeline agents: a deterministic router that hands all domain work to 14 specialist subagents. An InfoQ article the same day frames agents as the next step after microservices, with decisions rather than functions as the unit of decomposition.
• Shopify (Oct 21) described how it post-trains open models as “task experts” for Sidekick subagents. For one subagent, a post-trained model scores slightly higher than the frontier model at about 90% lower serving cost.
• llm-d 1.0 (Oct 20). Google and the llm-d community released a Kubernetes-native serving stack with cache-aware routing, tiered KV cache and flow control for bursty agent traffic.
• AWS and vLLM (Oct 19). Bi-directional KV transfer lets disaggregated serving reuse earlier turns instead of recomputing them.
• The Pragmatic Engineer (Oct 20) gathered AI spending data from 27 companies and found that few can yet say what the spending returns.

Why it matters. Per-token prices keep falling. MiniMax’s M3.1 Flash (Oct 19) is listed at $0.15 per million input tokens. But an agent’s bill grows with the number of calls and the size of the context resent on each call. The posts this week treat cost as a property of the agent’s design, not just of the model’s price list.

Key ideas

• Measure where tokens go first. Before its changes, Cloudflare’s review spend split into 43% cache writes, 34% cache reads and 23% output tokens. Most of the money went on context the model had already seen. Tokens per review had more than tripled since April, even with a 97% cache hit rate.
• Check the first call of each session. Cloudflare found that the first request in a session read a cached prefix only 14–26% of the time. The shared agent prompt and per-run details such as the working directory and date sat in one cached block, so the shared part was never stored on its own. Splitting that block, removing wasted calls and sharing one prefix across three reviewers produced about 40% of the savings. Model and effort changes, tested in randomized splits, produced the rest.
• Cheaper settings need a quality check. Cloudflare had judges from two providers rate a blinded sample of 657 findings. Code quality at high effort found about as many real problems with less noise. Medium effort lost real findings for code quality and performance, so code quality is moving to high effort and the other reviewers are being tested at high next.
• Keep the router thin. Meta’s first design, one agent loaded with all the domain knowledge, began applying one subsystem’s conventions to another once its context grew. The replacement is a 1,300-line orchestrator that follows a fixed seven-step protocol; only the delegation step uses LLM reasoning, inside a subagent that loads just its own knowledge files. Configuration prep for a new data column fell from three to four days by hand to under an hour. “Which subagent handles this?” is a routing decision, the post argues, so an LLM should not make it.
• Bound the loop. InfoQ’s article (Mallika Rao) warns that evaluation-driven retries, unlike retries on explicit errors, are hard to bound and can raise latency and backend load. It recommends latency budgets, retry limits, limits on reasoning depth and deterministic fallback paths.
• Narrow tasks suit small post-trained models. Shopify’s GraphQL subagent reads about 40 tokens for every one it writes. A post-trained Nemotron 3.5 Lightning (3B of 30B parameters active) went from a judge score of 37.7 to 74.5 over six loops, against 73 for the frontier model. It responds 7× faster at the median. A nightly “self-healing” step repaired 69% of the failed production conversations it tried and fed them back as training data.
• Route each step to the replica that holds its context. llm-d’s router sends requests to replicas that already cache the prompt prefix and spills KV cache to host memory, SSD or Managed Lustre. On an agentic coding workload on Ironwood TPUs, Google reports more than 85% lower median time to first token and 2.4× throughput compared with a plain Kubernetes Service.
• Don’t recompute earlier turns. In disaggregated serving, KV cache normally flows only from prefill to decode nodes. AWS’s vLLM change lets decode nodes send it back on the next turn, cutting time to first token by up to 1.6× for Qwen3-32B and 3.3× for GLM-5-FP8 at the longest prompts tested. Below a configurable threshold, recomputing is cheaper than transferring.

Composite of the Cloudflare, Shopify, llm-d and AWS write-ups; not any one source’s diagram. Tap to enlarge.

Analysis: the cheapest change in these reports is often the most boring one, such as prompt ordering, before any model swap. Meta reports time saved rather than token costs, but a router that does no LLM reasoning and specialists with small, isolated contexts are what allow each step to get its own model, as Shopify does for its subagents. Teams that self-host gain the serving-side levers (routing and KV reuse); teams on hosted APIs mostly have the caching and model-choice levers. Open question: the Pragmatic Engineer data suggests most companies still cannot tie AI spending to outcomes, which makes per-step cost the easier number to manage than return.

Reading list

Grouped loosely: application-side savings (Cloudflare, Shopify), orchestration patterns (Meta, InfoQ), serving-side savings (llm-d, AWS) and spending context (The Pragmatic Engineer, MiniMax, Google Cloud).

• Cutting our inference bill by 68% by being obsessive about optimization — Cloudflare, Oct 19, 2026. Where review tokens went, the cache-prefix fix, model and effort experiments, and a careful quality analysis with acted-on findings and blinded judges.
• How Shopify turns open models into task experts — Shopify Engineering, Oct 21, 2026. The full loop: rubric and judge, pushing the frontier agent first, picking an open model, serving benchmarks, self-healing data and post-training.
• The Thin Orchestrator — Engineering at Meta, Oct 19, 2026. Why one all-knowing agent started mixing up conventions, the seven-step protocol, and how workflow templates reuse the same 14 specialists for new use cases.
• From Microservices to Agents: the Next Evolution of Distributed Systems — InfoQ, Oct 19, 2026. Planning separated from deterministic execution, specialised agents, evaluation inside the loop, and failure modes such as silent quality degradation and retry amplification.
• Announcing llm-d 1.0 — Google Cloud, Oct 20, 2026. What 1.0 adds (flow control, session-aware routing, safer rollouts), how GKE Inference Gateway runs the router, and reported results from Google and customers.
• Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM — AWS Machine Learning Blog, Oct 19, 2026. The reverse KV path, the recompute threshold, benchmarks on P5en instances, and how to turn it on in vLLM v0.21.0 or later.
• What’s the ROI of software engineers’ spending on AI? — The Pragmatic Engineer, Oct 20, 2026. Spending ranges, hard caps and PR-throughput data from 27 companies, plus CodeRabbit and Gitar aggregates. Paid post; this summary is based on the free preview.
• MiniMax M3.1 Flash shows China’s AI price race has no floor in sight — The Decoder, Oct 19, 2026. Pricing and Artificial Analysis scores for MiniMax’s new model, including a 93% hallucination rate because it rarely declines to answer. MiniMax’s announcement (Oct 19) has the vendor benchmarks.
• Who judges the judge? Three helpful ways to audit your LLM evaluators — Google Cloud, Oct 19, 2026. Judge biases and three checks (stability, agreement with human experts, multiple judge models) worth running before using a judge to approve a cheaper setting.

Background and prior art

• Prompt caching — Anthropic documentation. How prefix caching, cache breakpoints and write/read pricing work, which explains why prompt order mattered so much in Cloudflare’s case.
• Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., Sep 2023. The vLLM paper behind today’s KV cache management.
• DistServe: Disaggregating Prefill and Decoding — Zhong et al., Jan 2024. The case for splitting prefill and decode, which the AWS change and llm-d build on.
• Earlier in this series: Measuring AI-Assisted Engineering (Oct 17) on why AI spend is hard to tie to outcomes.

Enjoyed this? Follow me on X or connect on LinkedIn.