Post

Measuring AI-Assisted Engineering

DX data shows PR throughput up 75.5% while first reviews wait longer; why AI code percentage is a lens, not a KPI, and where saved time goes.

Posted
By
6 min read
Measuring AI-Assisted Engineering

AI coding tools have raised output, and the measurement question has moved to what happens around the code. Two DX issues from Oct 7 and Oct 14 and a Pragmatic Engineer report from Oct 15 show where the gains land and which numbers mislead.

Part of Weekend Deep Dive #1: Agent-Scale Infrastructure.

What’s new this week

  • The gap between code velocity and trust (DX, Oct 14). DX sells engineering-metrics tooling, and TrueThroughput is its own metric. Grace Fu and Scott Carey summarise DX’s Q3 2026 State of AI Impact in Engineering report. According to DX, median weekly TrueThroughput (its complexity-weighted PR throughput metric) is up 75.5% over the last four quarters, while organisations with the highest AI code volume waited 105% longer for a first review than those with the lowest. Median PR size is up 122% year over year.
  • AI Code Percentage as a lens, not a KPI (DX, Oct 7). Brian Houck argues that the share of AI-authored code has outlived its role as an adoption proxy. Per DX, weekly active AI usage now exceeds 95% across more than 500 companies, and the AI-authored share of merged code rose from 19% in Q3 2025 to 52% in Q2 2026.
  • Internal vibe-coding apps (The Pragmatic Engineer, Oct 15). A free bonus issue, first sent to paying subscribers a week earlier, in which Gergely Orosz describes Ramp’s Glass and Stripe’s Harbor: internal platforms on which non-engineers build their own tools.

Why it matters. The code review post in this series reports the Claude Code team measuring success in human time per change, with revert and incident rates as the guardrail; according to that team, the share of PRs with substantive review comments went from 16% to 54% when its review preview launched. DX’s industry data points the same way from the outside: generation is no longer the constraint, and the metrics that matter sit in review, verification and delivery.

Key ideas

Where the bottleneck moved (DX, Oct 14)

  • Throughput keeps climbing; the rest of the SDLC does not. Per the report, “what is struggling to keep up is the rest of the SDLC – the review, testing, and verification that turn code into something a team can trust.”
  • Two theories for slower first reviews. Beyond bigger PRs, DX suggests AI-generated changesets take more effort to review, or reviewers hesitate to pick up large, unfamiliar diffs. DX says its data cannot yet tell these apart.
  • Saved time goes to keeping systems running. In DX’s session telemetry, about 33% of AI sessions went to maintenance and 23.3% to fixes, against 11% on new capability. DX’s CAFE(S) framework treats much of this friction as a context problem rather than a model problem.

How to read AI code share (DX, Oct 7)

  • A good variable, a bad target. Houck’s test for any goal metric is whether gaming it still produces the wanted outcome. AI code share fails: a team can hit a quota by routing trivial changes through agents. Two teams at 60% can also work in very different ways, from autocomplete to delegating whole tasks.
  • Generated is not shipped. Houck proposes survivorship as a companion metric and cites a 2026 study of more than 200,000 code units across 201 open-source projects that tracked agent-authored code after merge.
  • Review it quarterly. AI code share describes a structural property that moves slowly; watching it weekly creates noise and a temptation to push the line up.

Adoption beyond engineering (Pragmatic Engineer, Oct 15)

  • Small builds, wide use. Per Orosz, Stripe’s Harbor was built by one engineer in about six weeks, and onboarding to Ramp’s Glass is now part of new-hire onboarding. Reasons he lists include safe access to internal data and the company’s own design system.
  • Build versus buy. Orosz argues that building such a tool can take about as much effort as selecting a vendor once compliance reviews and price negotiations are counted, and that it trains engineers in AI infrastructure. He expects more companies with their own coding-agent harness to follow.
flowchart LR
    U["AI usage and AI code share<br/>(segmenting lens, DX Oct 7)"] --> T["PR throughput<br/>+75.5% in four quarters"]
    T --> Z["Median PR size<br/>+122% YoY"]
    Z --> W["First-review wait<br/>+105% at highest AI volume"]
    W --> D["Perceived ease of delivery<br/>+3% QoQ"]
    G["Guardrail: revert and incident rates<br/>(Claude Code team, Oct 15)"] -.-> D

Figures from DX’s Oct 14 issue except where noted; the arrows are an editorial reading of those figures, not a measured causal chain.

Analysis: taken together, the sources suggest a measurement set rather than a single number: throughput, PR size, time to first review, revert and incident rates, and time allocation between maintenance and new work, with AI code share used to segment teams rather than as a goal. Internal builder platforms like Glass add a population the usual engineering metrics do not cover at all.

Open question: DX reports that code review is among the smallest AI use cases in its telemetry. Whether agent-assisted first-pass review, of the kind the Claude Code team describes, shortens the review wait at industry scale is not yet in the data.

Reading list

  • The gap between code velocity and trust is growing — DX Engineering Enablement, Oct 14, 2026. The headline findings of the Q3 report: throughput, review waits, PR size, the Developer Experience Index breakdown and session telemetry, with two recommendations. The full report is linked from the post.
  • Don’t measure AI Code Percentage on its own — DX Engineering Enablement, Oct 7, 2026. The case for treating AI code share as an independent variable, with the gaming test, the survivorship funnel and practical guidance on review cadence. It maps the questions onto DX’s Core 4 dimensions.
  • The Pulse: new trend of building “internal vibe-coding apps” — The Pragmatic Engineer, Oct 15, 2026 (free issue). A first-hand look at Ramp’s Glass and a summary of Stripe’s Harbor, with six reasons companies build rather than buy. Ramp has since launched Glass as an external product.
  • How the Claude Code team does code review — claude.dev, Oct 15, 2026. Covered in the code review post. One team’s measurement model: human time per change, revert and incident rates, and the 16%-to-54% change in substantive review comments.

Background and prior art

AI & Agents, Weekly Deep Dive
developer-productivity strategy
This post is licensed under CC BY 4.0 by the author.