Trust the Eval Before the Model: Benchmarks, Labels, Spend and the Model Menu

The model menu changed again between Oct 23 and Oct 29, and the main public index changed its own tests in the middle of it. Two eval post-mortems from the same week show that labels and judge prompts can decide a model comparison before any model is run, and a third-party measurement of Claude Code shows how much spend can hide in calls nobody asked for.

Part of This Week in AI & Systems #3: Rules in Code, Controls in IAM.

---

What’s new this week

• Artificial Analysis Intelligence Index v5 (Oct 27). The index now draws 60% of its score from private or held-out test sets.
• Gemini 4 Carbon (Oct 28). Google DeepMind’s new frontier model, announced two weeks after Argon and released to selected security teams first.
• Kimi K3.1 Pro (Oct 23). Moonshot’s multimodal MoE with 2.8T total and 104B active parameters, a 1M-token context and open weights promised by Nov 4.
• GPT-6.1 Terra (Oct 26). OpenAI’s mid-tier model for computer use at $1 per million input tokens ($0.05 cached) and $5 per million output tokens.
• Eval quality (Oct 26–27). Property Finder (via AWS) found copied labels, Ai2 measured judge-prompt sensitivity, and Netflix described Pensive AI, an LLM on top of deterministic failure diagnosis.
• Claude Code’s status line (Oct 29). A third-party measurement attributes about 16% of estimated spend to background status-line summaries.

Why it matters. Choosing or swapping a model is only as good as the evidence behind it. This week shows three places that evidence can be wrong: the public index, the team’s own labels and judges, and the spend data.

Key ideas

Benchmarks move under the models

• Mostly private, merge-graded. Index v5 weights Knowledge Work, Coding, Science and General at 25% each. It adds Cognition’s FrontierCode 2.0 (132 tasks in real repositories, graded on whether a maintainer would merge the change) and Terminal-Bench-Science, and replaces HLE with HLE-Diamond.
• The open-weights gap widened. On v5, Claude Opus 5.5 (max) scores 58, GPT-6 Astra 57, GPT-6.1 Sol 55 and Gemini 4 Argon 50. Artificial Analysis says the gap to the best open-weights model widened from 3 points in v4.3 to 12, measured against Kimi K3.1 Pro, whose weights are not yet released. The best models with released weights score 32, below Claude Haiku 5.5 at 36.
• Scores depend on the index version. At its Oct 23 launch, Kimi K3.1 Pro scored 55 on the previous version of Artificial Analysis’ index, behind Opus 5.5 (58) and ahead of GPT-6 Astra and Gemini 4 Argon (53 each). Four days later, on v5, it scores 46. Comparing a launch-day score with a later one only works on the same index version.

The model menu

• Gemini 4 Carbon. Carbon scores 59 on Index v5, up from Argon’s 50, and leads SEC-bench Pro (88.2%) and ExploitGym (54.8%), with 71.0% on CWE-bench for patching. Google reports that cheating on Terminal-Bench 4.0 and the indirect prompt-injection attack success rate both fell to 0.0%. The Decoder adds that Carbon still trails Claude Opus 5.5 on the coding-agent index, 59 to 62, and uses 2.3 times as many reasoning tokens as Argon. Access starts with selected security teams; the introductory price matches Argon at $2 per million input tokens and $10 per million output tokens.
• Kimi K3.1 Pro. Priced at $2/$10 per million tokens, half the price of Opus 5.5, with a Highspeed tier that is 2 to 3 times faster at twice the price. The gains come mostly from far more RL post-training, and the launch includes Kimi Hosted Agents, a managed 24/7 harness.
• GPT-6.1 Terra. On Agents’ Last Exam at medium effort, Terra scores 57.6% at $0.29 per task, the same score as GPT-6 Astra at $4.82 per task. On OpenAI’s internal set of error-inducing prompts, the low-effort error rate fell from 53.6% to 16.7%. Coding lags: 50.3% on Terminal-Bench 4.0 against 70.6% for Claude Sonnet 5.5, and OpenAI still recommends Sol or Astra for the hardest coding work. All scores are OpenAI-reported. On Index v5, Terra is the cheapest of the top ten at $0.96 per task.
• A rule of thumb from OpenAI. The Decoder relays Ted Sanders’ advice: “pick Astra” for quality, Sol if somewhat price-sensitive, and Terra or Luna if very price-sensitive.

Check the eval before the model

• Where did the labels come from? When Property Finder tried to replace its vision model, it found that 96.8% of its evaluation labels had been copied from the incumbent model’s predictions, so every alternative was marked down for disagreeing with it. A domain expert labelled 356 images from scratch, single labels became multi-label, and Micro F1 became the main metric. After 27 prompt versions, later with automated evolutionary optimisation, the team chose a model on Bedrock that was more accurate across 11 image attributes at 86% lower cost.
• Test the judge prompt. Ai2 found that changing a few words in a judge prompt can move its accuracy by more than 50 percentage points, and that retrieval and extra reasoning did not reliably help.
• Deterministic first, LLM second. Netflix sees roughly 100,000 failed data and ML workloads on peak days, and half of Pensive’s rule-based classifications came back “unknown”. Pensive AI keeps the rules and asks an LLM only for the first and last line numbers of an error span, then rebuilds the text from its own copy of the log, so the model cannot invent log lines. It now extracts errors from 100% of failed workloads. Frozen legacy rules serve as regression labels, logs are archived to S3 so evals can be repeated, and eval thresholds gate merges and promoted builds.

Know what the harness spends

• Calls nobody asked for. A write-up by Flamerix AI says Claude Code sends each background agent’s full transcript to that agent’s model about every 30 seconds to get a 3-to-5-word status line. Grouping Claude Code’s OpenTelemetry events by query source, the authors attribute 16.4% of estimated spend and almost 20% of API requests to these summaries, more than the main conversation itself. They link four other measurements that found 18% to 26%, and publish an unofficial preload hook that suppresses the timer without modifying the binary. This is a third-party measurement; Anthropic has not commented in the sources found.

Composite of the Property Finder, Ai2 and Artificial Analysis write-ups; not any one source’s diagram. Tap to enlarge.

Analysis: before a model swap, the cheaper check is often the eval itself: where the labels came from, how sensitive the judge is to wording, and which index version a score refers to. Netflix’s design shows the same discipline in production, with a bounded model output and evals that gate releases. Open question: Carbon’s agent-reliability claims, such as the 0.0% figures, come from Google and will only be testable once access widens beyond security teams.

Reading list

Grouped loosely: benchmarks (Artificial Analysis), models (Google, The Decoder, Latent Space, Simon Willison, OpenAI), eval quality (AWS, Hugging Face / Ai2, Netflix) and harness spend (Flamerix AI).

• Announcing Artificial Analysis Intelligence Index v5 — Artificial Analysis, Oct 27, 2026. The new weighting, the private and held-out sets, the added benchmarks and the full leaderboard.
• Introducing Gemini 4 Carbon — Google, Oct 28, 2026. Security benchmarks, reliability claims, access and pricing. The Decoder (Oct 28) adds the coding-agent comparison and token usage; Latent Space’s AINews (Oct 29) collects reactions.
• Introducing Kimi K3.1 Pro — Simon Willison, Oct 23, 2026. A hands-on first look, with pricing. Latent Space’s AINews (Oct 24) covers the launch-day scores and Hosted Agents.
• Introducing GPT-6.1 Terra — OpenAI, Oct 26, 2026. Computer-use, factuality and coding results with pricing. Simon Willison’s notes and The Decoder (both Oct 26) add context.
• OpenAI’s Ted Sanders shares a rule of thumb for picking the right GPT model — The Decoder, Oct 27, 2026. A short guide to OpenAI’s current tiers.
• Fix evaluation bias to unlock better models on Amazon Bedrock — AWS Machine Learning Blog, Oct 27, 2026. How Property Finder found the copied labels, rebuilt the ground truth and iterated on prompts.
• Can we trust LLMs to judge whether a research idea is novel? — Hugging Face / Ai2, Oct 27, 2026. The novelty-judge benchmark and its prompt-sensitivity results.
• Making failure diagnosis reliable with Pensive AI — Netflix TechBlog, Oct 26, 2026. Line-number outputs, shared deadlines, per-attempt caching and evals in CI. The page blocks automated fetching; this summary is based on the RSS full text.
• Claude Code burns ~15-20% of your tokens on a 3-word status line — Flamerix AI (GitHub Gist), Oct 29, 2026. The method, the per-source breakdown and the unofficial hook. A third-party measurement.

Background and prior art

• Earlier in this series: Agent Verification (Oct 17) on evals that keep running after launch, and The Cost of an Agent Step (Oct 24), where Cloudflare and Shopify used judges to check that cheaper settings kept quality. Google Cloud’s three judge audits linked there (stability, agreement with experts, multiple judges) are a direct companion to Ai2’s results.

Enjoyed this? Follow me on X or connect on LinkedIn.