Don’t Panic: A Hitchhiker’s Guide to Agentic DevOps
A presentation at KCDC 2026 by Jeremy Meiss
Don’t Panic A Hitchhiker’s Guide to Agentic DevOps Jeremy Meiss
Jeremy Meiss Tech Solution Architect · WWT DevOpsDays KC organizer CommunityDays KC organizer CDF Ambassador 2025-26 World Wide Technology ☀ 1 / 55
“ AI is an amplifier. DevOps Research and Assessment (DORA), 2025 n = 5,000 World Wide Technology ☀ 1 / 55
CI throughput increase is deceptive +59% Top 5% 28M workflows analyzed - CircleCI 2026 median team +4% · bottom quartile +0% CI throughput, YoY CI throughput increase is deceptive where almost all of it sits ☀ 1 / 55
Our three takeaways
- Why some teams are pulling ahead while most aren’t 2. What the actual bottleneck is now 3. A set of steps you can bring to your VP
“Mostly Harmless”
Where teams actually sit 01 02 03 AI-Assisted AI-Optimized Copilot, Cursor, ChatGPT for the odd question. 72% use it daily (Sonar 2026). Four tools per team, no governance. An actual decision got made - strategy, tool selection, security review. Roughly the top quarter. AgentAugmented Where teams actually sit Agents doing real end-toend work. The human moves from in the loop to over the loop. The top 5%. ☀ 1 / 55
✋ World Wide Technology ☀ 1 / 55
It isn’t a straight line World Wide Technology ☀ 1 / 55
The J-Curve Dip Learning curve Verification tax Pipeline adaptation The tuition cost, not the failure. World Wide Technology ☀ 1 / 55
Three reports, three conclusions Google Cloud Stanford AI Index MIT NANDA OPTIMISTIC NEUTRAL PESSIMISTIC 78% of executives have seen ROI on at least one use case. Adoption is everywhere. Real structural change is rare. The shadow AI economy - official tools underdeliver, people route around them. World Wide Technology ☀ 1 / 55
DORA (DevOps Research and Assessment) DORA’s 2026 AI ROI report under fire. Separately: Stride Research has pre-registered a study testing whether AI adoption still correlates with performance once you control for company size. Results expected later this year - not published yet. World Wide Technology ☀ 1 / 55
“ Be as skeptical of the AIoptimism industrial complex as you are of the AI-hype industrial complex. World Wide Technology ☀ 1 / 55
It isn’t just DORA Atlassian, independently this year: the AI efficiency paradox - individual output speeds up, then backs up at review and approval. Different company, different survey. No connection to DORA. World Wide Technology ☀ 1 / 55
Not a survey 28,000,000 CI workflows analyzed CircleCI 2026 - not a survey, actual pipeline data Not a survey ☀ 1 / 55
Throughput growth by percentile +97% +47% +25% Top 5% Top 10% Top 25% +4% flat Median Bottom 25% CircleCI 2026 World Wide Technology ☀ 1 / 55
The bottleneck moved Feature branch Main branch +85% +50% +26% +15% flat -7% Median Top 10% Top 5% CircleCI 2026 World Wide Technology ☀ 1 / 55
Slower at fixing what breaks 72 min Median time to recover +13% YoY · feature branches: 80 min (+25%) - CircleCI 2026 Slower at fixing what breaks ☀ 1 / 55
Main-branch success rate 70.8% 76.7% Lowest in 5+ years · benchmark is 90% Improving - still short of the 2023–24 mid-80s and the 90% benchmark September 2025 Main-branch success rate March 2026 ☀ 1 / 55
Merge Efficiency Ratio (MER) - validation cycles per merged change 3.9 2.6 1.3 Median team Top 5% Elite cohort 20 organizations · CircleCI, July 2026 Merge Efficiency Ratio (MER) - validation cycles per merged change ☀ 1 / 55
50 dev team, ~3k changes a month $900K $700K+ Unoptimized workflows - agents idling until their context cache expires By moving routine validation earlier - CircleCI, July 2026 a year at risk 50 dev team, ~3k changes a month recoverable ☀ 1 / 55
“ Throughput is a vanity metric. The real metric is integrations that ship. World Wide Technology ☀ 1 / 55
Tokenmaxxing Claudeonomics leaderboard - “Token Legend” tier Meta: 85,000 employees. 60 trillion tokens. One month. Uber: entire 2026 AI budget gone. Month 4 of 12. Amazon, weeks later: leaderboard deleted. “Don’t use AI just to use AI.” Reported across multiple industry outlets, 2026 - figures not independently verified. World Wide Technology ☀ 1 / 55
The gap is where incidents come from 96% don’t fully trust AI code 48% always verify before committing Sonar 2026, n=1,149 developers The gap is where incidents come from ☀ 1 / 55
“ Code that looks correct but isn’t reliable. 61% of developers Sonar, 2026 World Wide Technology ☀ 1 / 55
Predicted, felt, measured Predicted: +24% faster Felt: +20% faster Measured: 19% slower METR, randomized controlled trial, July 2025 - 16 developers, 246 real tasks. World Wide Technology ☀ 1 / 55
2026: they tried to re-run it METR tried to re-run the study in early 2026 to see whether newer tools changed the result - and the study itself broke. Too many developers refused the “no AI” condition, even for pay. There was no longer a clean control group left to recruit. World Wide Technology ☀ 1 / 55
“ AI proposes. Humans dispose. Every AI output is a junior dev’s first PR. World Wide Technology ☀ 1 / 55
Seven archetypes, six groups C1 C2 C4+C5 C6 C7 10% 11% 17% 22% 20% 20% Foundatio nal chal‐ lenges High-impact & stable teams Pragmatic performers Harmonious highachievers The legacy bottleneck C3 Constrained by process DORA 2025, n=5,000 World Wide Technology ☀ 1 / 55
Add it up 38% 40% clusters C1 + C2 + C3 clusters C6 + C7 - DORA 2025, n=5,000 in clear difficulty Add it up high-performing ☀ 1 / 55
The shadow AI economy 4 AI tools per team 35% access via personal accounts ChatGPT 52% · Perplexity 63% - Sonar 2026 The shadow AI economy ☀ 1 / 55
The experience gap - Sonar 2026 Juniors Seniors +40% productivity gain +32% productivity gain 66% report “looks correct but isn’t reliable” 48% report “looks correct but isn’t reliable” The experience gap - Sonar 2026 ☀ 1 / 55
“ AI cannot rescue an unhealthy platform.
If it can happen there… Dec 2025 Mar 2, 2026 Mar 5, 2026 Amazon’s fix Elevated permissions AI-assisted changes Worse Bypasses two-person approval, deletes and rebuilds the environment. 13-hour outage. 120,000 lost orders. 1.6 million website errors. Mandatory review, reinstated If it can happen there… ~6.3M lost orders, a 99% drop in North American order volume, ~6 hours down. Amazon disputes AI as the primary cause. Senior-engineer sign-off on AI changes. A human, back in the loop. ☀ 1 / 55
AI as an Amplifier works both directions. World Wide Technology ☀ 1 / 55
“ (Vibe) Build, then verify. Generate fast. Validate harder. World Wide Technology ☀ 1 / 55
Self-healing test automation Before After Coverage 30–50% Coverage 80–95% Authoring takes days to weeks Authoring takes hours Payback in 12–24 months Payback in ~3 months (vendor-reported figures) Self-healing test automation ☀ 1 / 55
Progressive delivery Push Change ships behind a flag → Small % of traffic Real users, limited blast radius Ship to 100% → Telemetry Error budget & p99 latency, watched live → Telemetry holds Autorollback Automatic — before anyone gets paged “the knack of flying lies in learning how to throw yourself at the ground and miss.” — Douglas Adams, The Hitchhiker’s Guide to the Galaxy World Wide Technology ☀ 1 / 55
The cost of skipping it +80% more outages, without a verification layer +46% more quality impact, without a verification layer Non-users vs. users of a deterministic verification tool - Sonar 2026 The cost of skipping it ☀ 1 / 55
The AI Capabilities Model - DORA 2025 01 02 03 Clear AI stance Healthy data ecosystems AI-accessible data A stated, communicated position on AI use. Data that’s well-governed at the source. Internal data AI tooling can actually reach. 04 05 06 Small batches User-centric focus Working in small, reviewable increments. Genuinely centered on the user, not the roadmap. Quality internal platforms World Wide Technology Platforms worth building on. ☀ 1 / 55
One protocol, any cloud AWS GCP Azure MCP Portable reasoning engine Zero-trust World Wide Technology Portable Auditable ☀ 1 / 55
“ Successful AI adoption is a systems problem, not a tools problem. DORA, 2026 World Wide Technology ☀ 1 / 55
The Roadmap
Three steps to take home 01 02 03 Pick a highvelocity, noncritical surface Instrument the right things Budget for the tuition, up front Main-branch throughput, success rate vs. a 90% benchmark, recovery time vs. 60 minutes - not lines of code. As we just saw, vanity metrics can hide the paradox at trillion-token scale. Tell your CFO before the dip arrives that you expect one. Spend political capital holding the line, not explaining yourself after the fact. Not billing, not auth. Internal tools, a dev dashboard. Put the J-curve dip somewhere it can’t hurt you. Three steps to take home ☀ 1 / 55
Reinvest on purpose, not by accident By accident The “free headcount” DORA describes gets absorbed into more feature pressure You stay the implementer Reinvest on purpose, not by accident On purpose Reinvested in quality, security, and paying down technical debt You become the systems architect and quality governor ☀ 1 / 55
“ Code is not an asset. Code is a liability. Software Engineering at Google via DORA 2026 World Wide Technology ☀ 1 / 55
“ We don’t measure AI by the code it writes, but by the bottlenecks it clears. DORA, 2026
Go clear the bottlenecks AI can’t.
Thank You Jeremy Meiss jerdog.dev /in/jeremy-meiss jerdog jmeiss.me Slides: https://speaking.jmeiss.me
We’ve all seen the demos where an AI writes a “To-Do” app in thirty seconds. It’s impressive, sure, in the same way a dog riding a bicycle is impressive - but it’s not exactly how we build production-grade software. In the real world the SDLC is messy, our pipelines are held together by legacy YAML, and “moving fast” usually means “breaking things we didn’t know existed.”
This talk looks past the hype and into the data: DORA, CircleCI, Sonar, and METR’s own randomized controlled trial, plus what happened when Meta, Uber, and Amazon took agentic AI further than almost anyone else this year. AI is an amplifier - it makes strong teams stronger and struggling teams worse, faster. The question isn’t what AI can build. It’s how you ship this without making everyone furious.
At the end, we’ll have a roadmap: what to measure, what to budget for, and why the most important part of AI-driven DevOps isn’t the code it generates, but the bottlenecks it actually clears.
