What is the actual cost of software failures?
Research key findings
Summary
What is the real cost of persistent failures in Continuous Integration (CI) pipelines? That is what a recent research initiative conducted by a Cambridge Judge Business School MBA project in collaboration with Undo sought to answer.
Every company is a software company, the ability for software engineering teams to deliver high quality software at speed is the difference between companies that gain a competitive edge versus those that fall behind.
While the adoption of CI is on the rise, the number of software failures in backlogs are too.
Despite the best efforts of software engineering teams, there are too many situational factors outside of their direct control that can cause the software to fail, and not enough hours in the day to fix them all. The time spent finding, and fixing, those failures is costing enterprise organizations billions annually.
The research concluded three key findings:
Adoption of CI best practices is on the rise.
The research found that 88% of enterprise software companies have adopted CI practices, compared to 70% in 2015. As a result of implementing CI, 35% of respondents say they deploy code changes hourly. 75% of respondents say that CI has improved the deployment of code changes a great deal.
88% of enterprise software companies have adopted CI practices, compared to 70% in 2015.
Reproducing software failures is impeding velocity.
41% of respondents say getting the bug to reproduce is the biggest barrier to finding and fixing bugs faster; and 56% say they could release software 1-2 days faster if reproducing the failures wasn’t an issue – 16% say they could deliver at least 4 weeks faster.
Software engineers spend an average of 13 hours to find and fix a single failure in their backlog.
83% of survey respondents say they cannot keep their test suites green and that the average amount of time needed to find and fix one failure in their backlog is 13 hours.
Failing tests cost enterprises $61 billion annually.
According to the research, 26% of developer time is spent reproducing and fixing failing tests – which equals 620 million developer hours a year. The total value of salary spent on those hours adds up to $61 billion annually and equates to $1.2 trillion in enterprise value lost for shareholders a year.
Over 600 million hours is spent on debugging code in North America each year. This equates to $61Bn in salary costs alone.
Conclusion
Although CI adoption is becoming ubiquitous, test suites are still plagued by a growing backlog of failing tests. Failures in integration and automated tests cause bottlenecks in the development pipeline, and substantially increase engineering costs.
The study further suggests that reproducibility of failures is also a major blocker, finding that not being able to reproduce issues slows engineering teams down and prevents them from releasing software changes at pace.
To fully realize the benefits of CI, time travel debugging offers a solution. It enables engineering teams to reproduce and fix software bugs faster. By eliminating the guesswork in defect diagnosis, development teams are able to accelerate Mean-Time-to-Resolution (MTTR) – resulting in considerable cost savings.