Ignoring Dead Canaries in your CI Coal Mine
Failures that emerge in test suites and go unresolved are akin to stepping over dead canaries in a coal mine. Developers are disregarding telltale warning signs if the failing tests go unaddressed. As a result, costly problems are going to emerge when the code is pushed into production.
Proper implementation of continuous integration (CI) best-practices is the key to success for delivering quality software at velocity. However, to realize the full potential of CI, it requires a zero-tolerance approach to test failures – tests must turn green.
However, the real challenge is not going green, it’s staying green. Any failures represent new regressions that must be addressed. Failing to do so can result in undermining the effectiveness of testing practices. It also creates a tolerance for failures that results in a growing backlog, and an increase in technical debt when failures emerge in production.
“You get this background radiation of spurious failures,” said Undo CTO and co-founder, Greg Law. “It means you can’t spot new ones. Most people get themselves to a place where they are not green and then it is very hard to get back.”
During their keynote at this year’s DBTest event (co-located within the all-virtual SIGMOD 2020) Greg Law and software architect Mark Williamson illustrated how software engineering teams can prevent themselves from falling behind once they are able to go green.
Greg outlined an approach to not only identify the flaky, sporadic tests that emerge in test suites, but to address them before they pile up in the backlog. Mark then took the reins and illustrated how you can spot, record, and fix these test failures using Undo’s Software Failure Replay (SFR) platform, LiveRecorder.
To see the demo, please take a few moments to watch the full keynote session below.