As market leader in enterprise application software, SAP helps companies of all sizes and industries run better. From back office to boardroom, warehouse to storefront, desktop to mobile device – SAP empowers people and organisations to work together more efficiently and use business insight more effectively to stay ahead of the competition.
SAP applications and services enable more than 345,000 business and public sector customers in over 190 countries to
SAP HANA is a scalable, massively multi-threaded database that forms the foundation of SAP’s technology stack and its product portfolio. As such, it is the backbone of major businesses worldwide, making quality, stability and reliability a central focus for the SAP HANA team(2).
SAP approached Undo to integrate Undo’s record, rewind and replay technology into SAP HANA’s automated testing environment so that SAP developers can find and fix errors quickly and efficiently in the extensive SAP HANA codebase with the aim of improving code quality. Undo is a recognised leader in software debugging(3) and its suite of innovative software quality tools(4) gives developers precise visibility into exactly what their code did before it misbehaved or failed.
SAP HANA - a comprehensive approach to testing
In a concerted drive to improve software quality, SAP uses fuzz testing as part of their routine QA process. Fuzz testing is a technique in which randomised test behaviours are presented to the system under test, making it possible to catch corner case bugs that were not anticipated by the system’s designers. This approach allows developers to find and fix failures - even highly obscure ones - before their users encounter them.
The demanding fuzz tests that validate builds of SAP HANA exercise the database server with randomly generated simulated user workloads. Combined with internal and external consistency checks, this provides a means to discover errors that would not be revealed by more traditional testing approaches.
However, the resulting test failures proved challenging to diagnose, due to a set of factors that are familiar to modern software vendors:
Complex control flowDifficult to make inferences about how a failure unfolded
Huge code baseCollaboration across teams is essential to pinpoint a bug
Non-deterministic failuresDifficult to reproduce reliably in order to investigate the root cause
These traits intersect: the need for cross-team collaboration makes it vital to have a reliable reproducer of the problem so that developers can work together on the same faulty behaviour, each assisting with a part of the puzzle. However, the non-deterministic nature of many of SAP HANA’s test failures makes collaboration challenging for the SAP HANA team because the failures cannot be reliably reproduced on developers’ machines.
Before approaching Undo, SAP developers investigated test failures using three primary methods to help them identify the root cause of failures:
Analysing logs from failed runsThese helped to produce a partial picture as to why a failure happened but often did not capture enough of the right information for the root cause to be easily identifiable.
Reproducing failures on live systemsFor complex problems that need to be debugged within a running system, a developer had to reproduce the original failure on a live system, which for rare faults was a time-consuming and unproductive use of resources.
Developer collaborationWhen complex problems could not be solved using the above methods, several developers with specialist knowledge would work together to figure out the source of the problem. Without the ability to reliably reproduce a failure on more than one machine, the developers often didn’t see the same program behaviour
Introducing Undo’s Live Recorder
Live Recorder for Automated Test can record all or part of a Linux (or Android) program’s execution for subsequent replay and analysis. It captures an exact recording of why a test failed, allowing developers to go back in time to any instruction in the program’s history and view the contents of any location in memory and any register. Recordings can be shared among developers and analysed on a different machine to the one on which the error occurred, making triage and analysis of failures much quicker, easier and more effective(5).
From failure to resolution: how Live Recorder works.
SAP identified Undo’s Live Recorder as a solution to multiply the value of their test strategy, making test failure results actionable by “closing the loop” between the fault and a developer understanding its root cause(6). By sharing recordings, developers can analyse an identical copy of the original failure while collaborating on a fix.
Recordings can be analysed using Undo’s interactive reversible debugger, UndoDB which allows developers to step or run their recorded program backwards as well as forwards in time to home in on a point of interest(7). The SAP HANA team is able to navigate quickly to the root cause of a problem using the full functionality expected of modern debuggers (such as scripting, conditional breakpoints and watchpoints, full inspection of globals and locals, etc.) in both forwards and reverse execution.
Improving software quality in SAP HANA using Live Recorder
SAP HANA is feature-rich and includes millions of lines of code so the Undo team optimised Live Recorder to meet SAP HANA’s complex requirements. SAP HANA already has a sophisticated continuous integration suite which Live Recorder enhances, helping developers get the most out of the tools already at their disposal. With one simple command, Live Recorder can be activated, making it easy to use Undo’s record, rewind and replay technology with minimum changes to the SAP HANA team’s existing workflow. SAP developers chose to deploy Live Recorder in always-on mode to a subset of their testfarm hosts. This allows them to phase in recording in parallel to their existing workflow, with their previous testing routine continuing as normal.
With Live Recorder, automated testing within SAP HANA has been made considerably quicker, easier and more effective.
Live Recorder generates recordings of each and every test failure, helping developers find and fix defects as the software is being written. Failures no longer need to be replicated on the machine on which they originally occurred: multiple copies of the exact program execution which led to a failure can be shared within the SAP HANA team, allowing developers to collaborate when solving the most complex errors. With hundreds of developers working on the SAP HANA database across multiple countries, the SAP HANA team can overcome language, communication and time-zone barriers when fixing problems, further strengthening the team’s responsiveness to issues that appear in testing.
For SAP HANA developers, engineers no longer need to draw inferences from textual logs to fix the failure but instead have total visibility of exactly what their program did before it failed, making debugging quicker and easier while further boosting stability and code quality to the underlying SAP HANA database.
SAP’s use of Live Recorder reflects the importance of software quality to SAP HANA. With minimum disruption to their existing workflow, SAP developers are using Live Recorder to maximise the value provided by SAP’s existing testing suite, increasing its value by letting them directly review and debug the behaviour of failing tests while the software is under development. The time and effort spent debugging is being reduced, freeing up time for writing new features.
Undo’s record, rewind and replay technology is becoming a central part of the testing process of SAP HANA. As companies like SAP move their products to the cloud, improving software quality and reducing time to fix issues that occur is becoming business-critical(8). SAP’s decision to use Live Recorder is an example of how software quality is helping SAP to maintain a competitive advantage in a volatile market environment, helping the teams to produce better quality code faster while improving productivity and collaboration between development and QA teams.
References and Related Links
(1) SAP: Corporate Fact Sheet
(2) SAP: What’s the business case for SAP HANA / SAP Blog
(3) Yahoo Finance: Software Quality Leader Undo Raises $3.3 Million in Series A Financing
(4) CIC: Undo’s Live Recorder Wins Award / CIC News
(5) Undo: Getting the most out of QA / White Paper
(6) Undo: SAP selects Undo / Press Release
(7) Undo: Reversible Debugging Whitepaper