Post

Developer Experience and DX Metrics

Developer experience is not a soft concern — it is the measurable output of a platform team. DORA showed that delivery performance predicts organisational performance.

Posted Updated
By
23 min read
Developer Experience and DX Metrics

Developer experience is not a soft concern — it is the measurable output of a platform team. DORA showed that delivery performance predicts organisational performance; the platform team’s job is to move every stream-aligned team up that curve.

Key Properties

Property Value
Canonical research baseline DORA Four Keys (Forsgren, Humble, Kim — Accelerate, 2018)
Multi-dimensional framework SPACE (Forsgren et al., ACM Queue, Feb 2021)
Practitioner DX framework DevEx / DX Core 4 (Noda, Forsgren, Storey et al., ACM Queue, May 2023)
DORA metric set (2024+) Throughput: deploy frequency, lead time, recovery time · Instability: change failure rate, rework rate
DORA 2025 shift Performance tiers replaced by seven team archetypes; AI Capabilities Model
Last tiered benchmark (2024) Elite 19%: on-demand deploys, < 1 day lead time, ~5% CFR, < 1 h recovery
AI-era finding AI amplifies existing strengths and weaknesses; still correlates with higher instability
Goodhart’s Law risk Any single metric becomes a target and stops being a good measure

When to Use / Avoid

Use When

  • You have a platform or enablement team and need to demonstrate value beyond ticket throughput.
  • You are setting quarterly OKRs for a platform team and need outcome-oriented KRs (not output-oriented ones like “deploy 5 features”).
  • You have enough delivery volume that the metrics are statistically meaningful — typically 10+ teams, 50+ deploys/month per service.
  • You want to create a feedback loop between platform investment and developer sentiment without waiting for a bi-annual survey.

Avoid When

  • You have fewer than 5 stream-aligned teams — measurement overhead exceeds signal.
  • You plan to use metrics for individual performance ranking — this guarantees gaming and destroys trust.
  • You do not have a plan for acting on the data — measuring without changing anything breeds cynicism faster than not measuring at all.
  • You are optimising for a single metric in isolation — always pair throughput metrics (deployment frequency, lead time) with stability metrics (CFR, recovery time).

DORA: From Four Keys to Archetypes

The four keys come from the Accelerate research (Forsgren, Humble, Kim, 2018): four years of survey data from 23,000+ respondents showing that software delivery performance predicts organisational outcomes — profitability, market share, customer satisfaction. The metric set has changed several times since, and the 2025 report changed how results are presented:

timeline
    title How DORA measurement evolved
    2014-2017 : Four keys defined
              : Deploy frequency, lead time, change failure rate, MTTR
    2021 : Reliability added as an operational metric
    2023 : MTTR renamed Failed deployment recovery time
    2024 : Rework rate added
         : Metrics grouped as Throughput vs Instability
    2025 : Elite/high/medium/low tiers dropped
         : Seven team archetypes and an AI Capabilities Model

The five delivery metrics today

Group Metric What it captures What it misses Typical gaming
Throughput Deployment frequency Batch size; how often value reaches production Whether those deploys work Counting staging deploys; splitting trivial commits
Throughput Lead time for changes Merge-to-production efficiency: review, CI, approvals Idea-to-commit time Small commits lower P50 while big ones sit in branches
Throughput Failed deployment recovery time How fast you restore after a bad deploy Blast radius Closing incidents early; re-categorising them
Instability Change failure rate % of deploys needing remediation Severity of the failure Silent rollbacks without an incident
Instability Rework rate % of deploys that are unplanned fixes for user-facing bugs Planned refactoring Labelling fixes as features

Pair throughput with instability, always. Measure at team and service level, never individual.

2024 benchmarks — the last tiered edition

Tier (share of respondents) Deploy frequency Lead time Change failure rate Recovery time
Elite (19%) On demand, multiple per day < 1 day ~5% < 1 hour
High (22%) Daily to weekly 1 day – 1 week ~20% < 1 day
Medium (35%) Weekly to monthly 1 week – 1 month ~10% < 1 day
Low (25%) Monthly to every 6 months 1 – 6 months ~40% 1 week – 1 month

Note the anomaly: medium performers had a lower change failure rate than high performers in 2024. DORA called change failure rate an outlier among the metrics — a reminder that a single number per tier hides a lot. The tiers come from cluster analysis of each year’s survey, so thresholds move year to year; do not treat them as fixed targets.

DORA 2025: archetypes and AI

The 2025 report (State of AI-assisted Software Development) replaced the four tiers with seven team archetypes, ranging from “harmonious high-achievers” to teams stuck with “foundational challenges” or a “legacy bottleneck”. Archetypes combine delivery metrics with well-being and product outcomes, so a fast team that is burning out no longer looks “elite”.

Two findings matter most for platform teams:

  • AI is an amplifier. AI adoption raises individual effectiveness and throughput, but still correlates with higher instability. Strong engineering systems turn AI into gains; weak ones get their weaknesses magnified.
  • Platform quality is one of seven AI capabilities. The DORA AI Capabilities Model lists a clear AI stance, healthy data ecosystems, AI-accessible internal data, strong version control, small batches, user-centric focus and quality internal platforms. The platform capability most correlated with a good user experience is “clear feedback on the outcome of my tasks”. DORA reports that around 90% of organisations now use at least one internal platform.

Source: DORA metrics history and the 2025 DORA report.

The SPACE Framework

Published in ACM Queue (February 2021) by Nicole Forsgren, Margaret-Anne Storey, Chandra Maddila, Thomas Zimmermann, Brian Houck, and Jenna Butler (Microsoft Research / GitHub / University of Victoria). SPACE’s central thesis: developer productivity cannot be captured by any single metric or dimension — choosing only one guarantees distortion.

graph TD
    SPACE[SPACE Framework] --> S[Satisfaction & Well-being]
    SPACE --> P[Performance]
    SPACE --> A[Activity]
    SPACE --> C[Communication & Collaboration]
    SPACE --> E[Efficiency & Flow]

    S --> S1[Developer survey scores]
    S --> S2[Burnout indicators]
    P --> P1[Reliability of delivered software]
    P --> P2[CFR / incident rates]
    A --> A1[PR volume, commit counts]
    A --> A2[CI runs, review turnaround]
    C --> C1[PR review participation]
    C --> C2[On-call rotation health]
    E --> E1[Lead time, WIP count]
    E --> E2[Interruption frequency]

Source: SPACE of Developer Productivity, ACM Queue 2021 — foundational paper establishing why multi-dimensional measurement is necessary.

The Five Dimensions in Practice

Satisfaction & Well-being captures how developers feel: job satisfaction, perceived productivity, burnout risk. Survey-based. Cannot be inferred from telemetry. This dimension detects problems before they show up in throughput metrics — a team’s deployment frequency stays stable for months before burnout-driven attrition collapses it.

Performance captures outcomes of the work: does the software reliably do what it is supposed to? CFR, reliability, SLO attainment. Not “did the team ship code” but “did that code work.”

Activity captures volume of work items — commits, PRs, deployments, code reviews. Useful for identifying flow blockers (a sudden drop in PR merges) but dangerous as a target (gaming is trivial and surveillance-adjacent).

Communication & Collaboration captures team interaction health: review latency, participation breadth in code review, documentation quality, on-call load distribution. A platform team should be watching whether their golden path reduces or concentrates coordination overhead.

Efficiency & Flow captures how smoothly work moves: WIP count, lead time, context-switch frequency, interrupt rate, time spent in meetings vs. deep work. Directly maps to cognitive load.

The SPACE recommendation: pick at least one metric from three or more dimensions. Never rely on a single-dimension slice.

The DevEx Framework and DX Core 4

The DevEx paper (Noda, Forsgren, Storey, Greiler — ACM Queue, May 2023) distills developer experience research into three actionable dimensions. The DX Core 4 (Noda and Tacho, 2024) operationalises these into a unified four-quadrant metric set that combines DORA, SPACE, and DevEx.

graph TD
    DX[DevEx Three Dimensions] --> FL[Feedback Loops]
    DX --> CL[Cognitive Load]
    DX --> FS[Flow State]

    FL --> FL1[CI duration]
    FL --> FL2[PR review latency]
    FL --> FL3[Test turnaround time]
    CL --> CL1[Number of systems to understand]
    CL --> CL2[Documentation completeness]
    CL --> CL3[Environment complexity]
    FS --> FS1[Unplanned interrupt rate]
    FS --> FS2[Context switch frequency]
    FS --> FS3[Meeting load per dev]

Source: DevEx: What Actually Drives Productivity, ACM Queue 2023 — Noda, Forsgren, Storey, Greiler — defining the three dimensions.

Feedback loops — the speed and quality of responses to developer actions. A CI pipeline that takes 45 minutes breaks flow. A PR that sits unreviewed for three days breaks flow. Fast feedback loops are the platform team’s most direct lever: invest in CI optimisation, parallelisation, test caching, and PR review culture before adding features.

Cognitive load — the total mental effort required to get work done. This is the Team Topologies lens applied to DX. Every system a developer must understand, every undocumented dependency, every environment they must manually provision adds to cognitive load. The golden path (see Golden Paths and Paved Roads) is a cognitive load reduction mechanism: it moves the load from the developer to the platform, which can bear it once for everyone.

Flow state — the capacity for deep, uninterrupted work. Research consistently shows that frequent interruptions destroy productivity non-linearly: an 8-hour day with 6 interruptions is not a 6-hour day, it is effectively 2. Platform teams should track interrupt rate (on-call pages per dev per week, Slack notification load, unplanned work ratio) as a proxy for flow state.

DX Core 4 Structure

The DX Core 4 organises metrics into four quadrants, each with one primary metric and three secondary indicators:

Quadrant Primary Metric Secondary Metrics
Speed Lead Time for Changes Deployment Frequency, CI duration, PR cycle time
Effectiveness Developer Experience Index (survey) Perceived productivity, friction index, tool satisfaction
Quality Change Failure Rate Deployment rework rate, incident frequency, SLO attainment
Impact % time on new capabilities Feature delivery ratio, toil ratio, platform adoption %

The DX Core 4 has been tested with 300+ organisations (as of 2024), with reported 3–12% increases in engineering efficiency and 14% increases in R&D time spent on feature development.

Survey-Based vs Telemetry-Based DX

These two measurement modes are complementary, not competing. Using only one produces a systematically incomplete picture.

graph LR
    subgraph Survey
        SU1[Captures: frustration, blockers, morale]
        SU2[Misses: what actually happened in pipelines]
        SU3[Risk: recall bias, social desirability]
    end
    subgraph Telemetry
        TE1[Captures: lead time, CFR, CI duration]
        TE2[Misses: why developers are frustrated]
        TE3[Risk: Goodhart's Law gaming]
    end
    Survey <-->|validate each other| Telemetry

Survey strengths: Captures subjective experience — frustration, perceived friction, morale — which telemetry cannot. A developer can tell you their local build takes too long; telemetry shows you the CI server time but not the local dev loop. Survey data detects problems 3–6 months before they manifest in delivery metrics. The DevEx framework explicitly recommends combining developer feedback with system telemetry.

Survey weaknesses: Recall bias — developers over-index on recent pain. Social desirability — developers underreport problems if they fear surveillance. Survey fatigue — quarterly pulse surveys lose response rates fast; short, frequent micro-surveys perform better. GetDX’s research shows that short monthly or even bi-weekly surveys with 3–5 questions outperform comprehensive annual surveys.

Telemetry strengths: Objective, continuous, cheap to collect at scale once instrumented. Lead time, deployment frequency, and CFR are unambiguous facts from VCS and deployment logs. No human recall required.

Telemetry weaknesses: Measures outputs, not experience. A fast CI pipeline that tests the wrong things looks excellent in the metrics. Easy to game once people know what is being measured. Telemetry tells you what happened; surveys tell you why developers find it painful.

When each lies: Telemetry lies when the system is gamed — teams split trivial commits to boost deployment frequency. Surveys lie when respondents fear negative consequences for honest answers — a sign of low psychological safety, which is itself a DX signal.

The operational recommendation: use telemetry to identify where the pain is (high lead time in environment provisioning), then use surveys to understand why (turns out: no self-service, always waiting on ops team).

Operational Metrics for the Platform Itself

The four keys measure how delivery teams are doing. The platform team needs a separate set of metrics measuring how the platform itself is performing. These are the SLOs of the platform-as-a-product (see Platform as a Product).

Metric Definition Target (indicative)
Pipeline success rate % of CI runs completing successfully without manual intervention > 95%
Median deploy time P50 time from deploy trigger to healthy rollout < 10 minutes
Environment provisioning time Time from self-service request to running environment < 15 minutes
Portal availability Uptime of the developer portal (Backstage, etc.) > 99.5%
Golden-path adoption % % of new services using the standard template/golden path > 80%
Time-to-first-deploy (TTFD) Time for a new service (using golden path) to reach its first production deploy < 1 day
Time-to-first-production for new developer Time from a new hire’s first commit to their first production deployment < 1 week
Toil ratio % of platform team time spent on unplanned, reactive, manual work < 20%

The time-to-first-deploy (TTFD) metric is especially diagnostic. If a golden path is genuinely working, a new service should reach production within a day. If it takes five days, the golden path has documentation gaps, broken tooling, or hidden manual gates that are not visible to the team building it.

Goodhart’s Law and Counter-Metrics

“When a measure becomes a target, it ceases to be a good measure.” — Charles Goodhart (1975)

Every DX metric is susceptible to gaming once it becomes an OKR target. The mitigation is structural: always pair metrics that trade off against each other so gaming one automatically degrades another.

The classic DORA trap: A team under pressure to improve deployment frequency deploys more often but without improving test coverage. Deployment frequency goes up; CFR goes up. The team looks productive on one metric while shipping more broken releases. This is not hypothetical — the 2024 DORA report found that rising AI adoption improved individual productivity measures while reducing delivery stability, and DORA’s leading hypothesis was larger batch sizes. The 2025 report still sees AI correlated with higher instability.

Pairing strategy:

If you measure… Also measure… Because…
Deployment Frequency Change Failure Rate High frequency + high CFR = faster failure shipping
Lead Time PR review coverage Short lead time + no review = just skipping process
DX survey satisfaction DORA throughput Happy devs on a slow platform = problem not surfaced
Golden path adoption % Lead time of adopters vs. non-adopters Adoption without benefit = platform not actually helping
Time-to-first-deploy Change Failure Rate of new services Fast first deploy + high early CFR = rushing people through

Anti-patterns to explicitly ban:

  • Lines of code — incentivises verbose code and discourages refactoring.
  • PR count — incentivises splitting work into many tiny PRs for appearance.
  • Ticket/story count closed — incentivises closing tickets prematurely or inflating estimates.
  • Individual developer rankings by DORA metrics — destroys psychological safety and collaboration. DORA metrics are team and system metrics, not individual performance metrics. The DORA research team is explicit on this point.
  • Developer keystrokes / active coding time — surveillance metrics that destroy trust and do not correlate with productivity.

Measuring AI’s Effect on Delivery

Most engineering organisations now run AI coding assistants or agents, and leadership wants to know whether they pay off. The same discipline applies: measure the system, pair metrics, and expect AI to amplify what is already there.

Layer Measure Watch for
Adoption Weekly active users of AI tools; share of PRs with AI involvement High licence count, low weekly use
Throughput Lead time and PR cycle time, AI-assisted vs not Faster merges with larger diffs
Instability Change failure rate and rework rate on AI-heavy changes Rework rising while throughput rises
Experience Survey: “AI saves me time”, cognitive load, trust in output Review fatigue on senior engineers
Platform Can agents use the golden path, CI and portal through APIs or MCP? Agents bypassing the paved road

The DORA 2025 conclusion is the one to take to leadership: AI gains depend on foundations — small batches, version control discipline, quality internal platforms. Buying more licences does not move the needle if the delivery system cannot absorb more change safely.

Worked Example: Platform Team OKR

A platform team in a scale-up with 8 stream-aligned teams, currently taking 5 days to get a new service from git init to first production deploy.

Objective: Reduce time-to-prod for a new service from 5 days to under 1 day.

KR1 — Adoption: 80% of new services created in Q3 use the Java golden path template.

  • Measured by: scaffold usage telemetry in Backstage.
  • Counter-metric: net promoter score of the golden path (developer satisfaction survey, 1 question, monthly).

KR2 — Speed: P50 lead time for services on the golden path is below 4 hours by end of quarter.

  • Measured by: VCS merge timestamp → production deployment event, filtered to golden path services.
  • Counter-metric: CFR for golden path services. If lead time drops but CFR rises, the path is cutting corners.

KR3 — Sentiment: Developer survey score for “it is easy to get a new service to production” is ≥ 4.0/5.0.

  • Measured by: monthly micro-survey (3 questions, 5-minute max) to all engineers, tracked by team.
  • Counter-metric: response rate. A response rate below 40% signals survey fatigue or fear of consequences.

What this OKR set avoids: It does not measure lines of code, PR count, or any individual developer output. It pairs every throughput metric with a quality or sentiment counter-metric. It sets a concrete outcome (5 days → 1 day) traceable to a platform action (golden path adoption) and validated by developer perception (survey score).

How Real Systems Measure DX

Dropbox — DX Core 4 at 1,000 Engineers

Dropbox’s engineering productivity team owns developer experience across roughly 1,000 engineers, covering CI/CD systems, telemetry infrastructure, and AI tooling rollout. In 2024, they adopted the DX Core 4 framework as their unified measurement model — the first major public case study of an organisation deploying DX Core 4 at scale.

The diagnosis: system metrics on foundational infrastructure (CI success rate, deploy time) were improving, but developer sentiment was not moving. The DX Core 4 revealed the disconnect: efficiency gains in the platform were not translating into perceived productivity because cognitive load from other sources (undocumented dependencies, unclear ownership) was absorbing the gains.

Their intervention: correlate survey data (DXI — Developer Experience Index) with system telemetry to identify which friction points developers actually noticed, rather than optimising for the metrics that looked best in dashboards. By 2025, they reported improvement across speed, quality, and impact quadrants simultaneously. AI adoption was scaled from one-third to three-quarters of engineers using AI coding tools weekly, achieved in about three months once the CEO began reviewing productivity data regularly — demonstrating that executive visibility is a DX forcing function.

Source: Dropbox uses DX Core 4 to define and measure engineering velocity

LinkedIn — Developer Productivity and Happiness (DPH) Framework

LinkedIn developed its own framework when an internal audit found over 50 separate dashboards and spreadsheets tracking developer metrics across engineering — with no shared definition of what productivity meant. The DPH Framework, open-sourced in 2023, organises metrics around developer activities (build, review, publish) and sentiment toward the tools used.

The central artefact is the Developer Insights Hub (iHub) — an internal product that visualises developer experience metrics for every team. Key innovation: the Developer Experience Index (EI), which converts continuous telemetry into a 0–5 scale with explicit thresholds. For local build time: < 10 seconds = score 5; > 5 minutes = score 0. This translates continuous metrics into actionable targets that product managers and engineers can reason about without needing to understand the full metric methodology.

The DPH approach surfaces Goodhart’s Law mitigation structurally: the index aggregates across multiple raw metrics, making it harder to improve the score by gaming a single underlying measure.

Source: LinkedIn DPH Framework — open-sourced measurement framework with goals, signals, and metrics for each developer activity.

Google / DORA — Four Keys Project

Google’s DevOps Research and Assessment team published the Four Keys project as an open-source BigQuery + Data Studio implementation that ingests deployment events, incident data, and VCS events to compute the four keys in near-real-time. The implementation pattern — using Cloud Functions to consume webhook events from GitHub/GitLab and a structured deployment frequency table in BigQuery — became the reference architecture for teams implementing DORA metrics without buying a dedicated tool.

The DORA team’s own internal guidance, applied across Google’s hundreds of product teams, uses deployment frequency and lead time as leading indicators and CFR plus recovery time as lagging quality checks. A team that improves DF without improving CFR is explicitly flagged as gaming the metric. The pairing is structural, not advisory.

Source: Using the Four Keys to measure your DevOps performance — Google Cloud Blog

Spotify — Backstage Developer Portal as DX Infrastructure

Spotify built Backstage not as a vanity developer portal but as the measurement and observability layer for developer experience. Every service in Backstage has an owner, SLO, CI status, and documentation completeness score. The software catalogue became the source of truth for golden path adoption: a service’s template origin is tracked, allowing the platform team to compute adoption rates and correlate them with delivery outcomes.

Scorecard plugins (Spotify’s Soundcheck, the open-source Tech Insights plugin) turn catalogue metadata into DX metrics: how many services have up-to-date runbooks, passing CI, valid SLO definitions. This operationalises the “golden path adoption %” metric without requiring a separate tracking system.

Spotify’s original driver was cognitive load. With 2,000+ microservices, the problem was not deployment automation — it was developers not knowing which services existed, who owned them, or how to deploy to them. The portal solved the cognitive load dimension of DX before the throughput metrics became the focus.

Source: How we use Backstage at Spotify — Pia Nilsson, Spotify Engineering Blog

Tools Landscape

Tool Primary Strength Primary Weakness Best For
GetDX Survey + telemetry in one platform; DX Core 4 native; strong research pedigree Expensive at scale; survey fatigue risk if not managed Teams wanting to combine sentiment + DORA in one place
Swarmia Clean DORA dashboards; lightweight; fast setup; Slack integration Less survey capability; shallower AI insights Teams wanting crisp DORA baseline without heavy rollout
LinearB Deep Git + Jira integration; workflow automation; predictive analytics Heavy to roll out; enterprise pricing; complex configuration Large engineering orgs with DevOps/platform resources
Jellyfish Engineering investment reporting (COGS, feature vs. debt); good for finance/execs Weak on raw DORA; expensive; more BI than DX tool Orgs needing R&D spend visibility alongside engineering metrics
Faros AI Flexible data connector (ingests from any source); open-source core Requires more configuration than SaaS alternatives Teams with heterogeneous toolchains needing custom pipelines
Four Keys (OSS) Free; BigQuery-native; good reference implementation (repository archived — use as a pattern, not a product) No UI; manual setup; no survey capability Budget-constrained teams comfortable with GCP
Backstage Tech Insights Native to Backstage; uses catalogue data; no separate tool Limited to catalogue-derivable metrics; no DORA out-of-box Teams already running Backstage wanting platform health scores

The recommendation for a platform team starting out: instrument a DORA baseline (Swarmia, or your own pipeline modelled on Four Keys) for telemetry baseline first. Add a survey layer (GetDX or Swarmia surveys) after you have 2–3 quarters of telemetry to correlate against. Do not buy a comprehensive platform before you know what questions you are trying to answer.

References

Software Architecture, Platform Engineering
team-building
This post is licensed under CC BY 4.0 by the author.