What a global pandemic can tell you about better DevOps practices
A presentation at Code PaLOUsa 2021 by Jeremy Meiss
DevOps & Software Delivery in a Global Pandemic
“
performance described vs performance derived
Jeremy Meiss Director, DevRel & Community
2 million 44,000+ 160,000+ 1,000x jobs/day orgs projects Larger than surveys
- 40k in 2019
- 150k in 2019 6
Four classic metrics Deployment frequency Lead time to change Change failure rate Recovery from failure time
CI/CD Benchmarks for high performance Throughput At will Duration <10 minutes Success Rate
90% Mean Time to Recovery <1 hour
The Data
Photo by: Matthew Henry
Throughput
Most teams are not deploying dozens of times per day
Image by Pawan Kolhe from Pixabay
Duration
Photo by Lukas from Pexels
Success Rate
Photo by Brett Sayles from Pexels
Recovery Time
Recovery Time
Recovery Time
The Insight
2020 has been a year.
Throughput
Throughput in a global pandemic
Peak Throughput was in April 2020
Duration
Duration in a global pandemic
Hypothesis: more tests written in March, driving up Duration. In April, a concerted effort on optimization
Success rate
Success rate in a global pandemic
Success rate in a global pandemic
Success rate in a global pandemic
Hypothesis: people working hard on core business stability
Recovery Time
Recovery time in a global pandemic
Hypothesis: few distractions* working at home
Important to set targets Throughput The average number of workflow runs per day Duration The average length of time for a workflow to run Mean time to recovery The average time between failures & their next success Success rate The number of successful runs / the total number of runs over a period of time Median CircleCI Developer Suggested Benchmarks 0.7 times/day Merge on any pull request < 4 minutes 5-10 minutes < 56 minutes Under 1 hour 80% for default branch 90% or better on default branch
Things that make you go 🤔
Branch information
No significant change in default branch from master… yet.
Success Rate on default branch higher than on non-default
Duration on default branches faster at every percentile
Recovery Time lower on default branches at every percentile
What development practices definitively work?
Success Rate does not correlate with company size
Duration is longest for teams of one
Recovery Time decreases with increased team size (up to 200)
Performance is better with >1 contributor
Software is collaborative
Language by Throughput
Language by Success Rate
Language by fastest TTR
Language by shortest duration
“Don’t deploy on Friday” is not a thing.
“Don’t Deploy on Friday” is not a thing ○ 70% less Throughput on weekends ○ 11% less Throughput on Friday (UTC) ○ 9% less Throughput on Monday (UTC)
Full Report https://circle.ci/ssd2020
Thank you.
In early 2020 a global pandemic set in motion events which none of us could predict. For us at CircleCI, data from 900k+ devs, 25K orgs, and 30m+ builds/mo provided us some interesting insights into better DevOps practices. In this session, we will seek to uncover what this unique dataset can tell us about high-performing engineering teams, and what trends stand out through the aggregated platform usage.
