Time travel debugging with rr debugger

Time Travel Debugging with Mike Shah

Today, you’re in for a treat! We have Mike Shah, Associate Teaching Professor, 3D Senior Graphic Engineer, and C++ conference speaker demonstrating how you can learn time travel debugging with the rr debugger.

Time travel debugging is used to understand what a program is doing and save you time when debugging. Over to Mike…

Time travel debugging is the idea of recording execution so you can “time travel” backward and forward through the program to find pesky bugs, or better understand the program.

We’ll be introducing you to some debugging tools, and working through an example so you can take these time travel debugging concepts to your own projects.

rr – Record and Replay

GDB has its own record and replay feature, but today we’re going to look at a lightweight Linux tool called “rr”, that enhances GDB and lets you record and replay the execution of the program with very little overhead.

Here’s an example program, replay.cpp:

#include <iostream>
#include <cstdlib>

int main() {
    int x;

    std::cout << “enter a value for x:”;
    std ::cin >> x;
    std::cout << “x is: “ << x << std::endl;

    srand(time(NULL));

    int randomValue = rand()% 42 + 1;
    std::cout << “A random number was generated:”
              << randomValue
              << std::endl;

    return 0;
}

A fairly basic example program, it creates an (uninitialized) variable x, asks the user for a value and repeats it to the user, then generates a random number and prints that as well. We’ll be going over this program with rr, which can be installed in the terminal with:

$ sudo apt-get install rr

and we’ll compile our example program (with debugging symbols, warnings, and errors enabled) using g++:

$ g++ -g -std=c++20 -Wall -Werror replay.cpp -o prog

We can run our test program here and make sure it’s worked:

$ ./prog
enter a value for x:100
x is:100
A random number was generated:37

This program is interesting to us as it’s non-deterministic. The user has to input a number and a random number is generated, so it is unlikely that the output of the program would be the same across runs.

Debugging a program that has the potential to change its behavior across multiple runs is where record and replay comes in handy. What rr does is to record in memory and on the disk the various states of program execution (for example, everything before main() is called, or when the different variables are set) while the program is running.

It’ll likely need administrator permissions and, depending on your system configuration, there may be certain flags needed to run rr, but it will tell you in an error output if you have configured it incorrectly.

Here, we need to use “sudo” and the “-n” flag, and we can run our program while recording:

$ sudo rr record -n ./prog
rr: saving execution to trace directory ‘/home/user/.local/share/rr/prog-3’.
enter a value for x:144
x is:144
A random number was generated:3

Now that we’ve recorded our program, we can look at the replay in rr (again here using sudo). Using rr replay you can inspect your most recent recording:

$ sudo rr replay

Now we’re greeted by the familiar start-up screen of GDB. We can use GDB commands like layout src and run to view the source code and start running the program. After continuing through the execution with continue you can see in the output that it’s showing you the same values that were input during the recording phase.

This is where time travel comes in. Let’s put a breakpoint somewhere interesting, like the line where we output the random number that was generated.

rr) br 19
Breakpoint 1 at 0x5611918a2275: file replay.cpp, line 19

Then run our program again using run then continue, and we’ll have reached our breakpoint. The functionality of rr is similar to GDB’s record and replay feature, and we can step backward with the same commands, so if you use reverse-next you’ll see we’re now at the line before our breakpoint.

This means that if you hit an error, like a segmentation fault, you can go back into the execution of the program and view the state before reaching the error. If we press “Enter” repeatedly, rr runs our most recent command (reverse-next) and we can get all the way back to the start of our main function, at which point we can use next to resume the execution normally; we can see that the output is the same as last time we ran through the program.

Running a Different Replay

When we recorded our program we saw that rr saved our recording in the “/home/user/.local/share/rr/” directory. If we inspect this directory we’ll be able to see all of our recordings through rr:

$ ls /home/user/.local/share/rr/
latest-trace prog-1 prog-2 prog-3

Here we can see all of the different recordings we’ve made, labeled with the program name and a number. If we wanted to inspect any of these in rr we would have to use:

$ sudo rr replay /path/to/file

Conclusion

The real value in rr and time travel debugging in general is the ability to rerun the same exact execution of a program, inspect it forward and backward, and even share the replays with your colleagues so they can help debug the particular run of the program.

Find more information about using rr at their website https://rr-project.org/ or their GitHub page https://github.com/rr-debugger/rr