Learnings on Better Software Delivery Principles Through a Panini

Pixabay via athree23 2

3

“

performance described vs performance derived

Jeremy Meiss Director, DevRel & Community

Dataset 257 mil+ 44,000+ 290,000+ 1,000x workflows orgs projects Larger than surveys 8

Four classic metrics Deployment frequency Lead time to change Change failure rate Recovery from failure time

CI/CD Benchmarks for high performance teams Suggested Benchmarks Throughput The average number of workflow runs per day Duration The average length of time for a workflow to run Mean time to recovery The average time between failures & their next success Success rate The number of successful runs / the total number of runs over a period of time Merge on any pull request 10 minutes Under 1 hour 90% or better on default branch

11

The Data

Photo by: Matthew Henry 13

Throughput the average number of workflow runs per day 14

Throughput

Throughput ~ Mar/Apr 2020 16

Throughput 17

Most teams are not deploying dozens of times per day

Goal: Continuous validation of your codebase via your pipeline

Image by Pawan Kolhe from Pixabay

Duration Image by Pawan Kolhe from Pixabay the length of time it takes for a workflow to run 21

Duration Image by Pawan Kolhe from Pixabay 22

Duration Image by Pawan Kolhe from Pixabay ~ Mar/Apr 2020 23

Duration Image by Pawan Kolhe from Pixabay 95th Median ~ Mar/Apr 2020 24

Photo by Brett Sayles from Pexels

Mean time to recovery average time between a pipeline’s failure and its next success

Mean time to recovery shortest MTTR ∝ Duration

“…the most robust — and certainly the fastest — solution to a broken build is to simply revert the offending commit, allowing troubleshooting to happen in a way that doesn’t interfere with the rest of the team. You can’t know whether a new build works or not unless you’re starting from a known good position, which means you should never allow a new build to start on a red build unless it’s explicitly designed to fix it, and it’s hard to imagine a commit more likely to fix a broken build than simply reverting the one that broke it to begin with.” - Brandon Byers, Head of Technology, NA @ Thoughtworks Photo by Brett Sayles from Pexels 28

Recovery Time ~ Mar/Apr 2020

Recovery Time ~ Mar/Apr 2020

Recovery Time ~ Mar/Apr 2020

Recovery Time ~ Mar/Apr 2020

Photo by Lukas from Pexels

Success rate The number of passing runs ÷ total number of runs over a period of time 34

Success rate ~ Mar/Apr 2020 35

Success rate ~ Mar/Apr 2020 36

Success rate ~ Mar/Apr 2020 37

Duration The average length of time for a workflow to run TTR The average time between failures & their next success 2019 (median) 2020 (median) This Year (median) Benchmark 3.38 min 3.96 min 3.7 min 5-10 minutes 52.5 55.11 73.6 min < 60 minutes 77% Average should be +90% on default branch 1.43/day As often as your business requires not a function of your tooling Success rate The number of successful runs / the total number of runs over a period of time 60% 61% Throughput The average number of workflow runs per day 0.80/day 0.70/day 38

Extra Insights

202x has been a year.

“Don’t deploy on Friday” is not a thing.

“Don’t Deploy on Friday” is not a thing ○ 70% less Throughput on weekends ○ 11% less Throughput on Friday (UTC) ○ 9% less Throughput on Monday (UTC)

Language shifts over the last few years 43

44

45

46

47

Vertical splits 48

Elite Performer validation 50th percentile on CircleCI fit into the “Elite performer” category on the 2021 State of DevOps report

2020 Report Full 2022 Report https://circle.ci/ssd2020 https://circle.ci/ssd2022 50

Timeline.jerdog.me Thank you. For feedback and swag: circle.ci/jeremy IAmJerdog jerdog /in/jeremymeiss