Can’t use a reversible debugger? Try these GDB commands.

In this GDB tutorial

In this GDB tutorial, we are reimplementing by hand what tools like LiveRecorder, rr, UDB, or even GDB’s built-in reversible debugging functions automate for you, using only GDB commands.

Why would you do that?

Sadly, full-featured reverse debugging is not always available (e.g. you’re running on an architecture or platform or program that rr can’t support, or you don’t have access to a LiveRecorder or a UDB license).

Let’s dive in.

 

Try UDB for free 
Find and fix test failures in minutes with UDB’s reverse debugging capability
Learn more »

 

Deterministic versus non-deterministic

First some theory because in this GDB tutorial, I am going to use an entirely deterministic program, so I can trace down step by step a bug in my code with only using GDB commands.

Deterministic software program:

• For a particular input, the program always gives the same output.
• Defects in the program are traceable because it follows a predictable execution pattern; you can determine the next execution step.

Non-deterministic software program:

• For a particular input, different runs of the program will give different outputs.
• Tracing defects in the program is complicated because you can’t determine the preceding or next steps of the execution due to more than one path the program can take.

Most of the software programs that we work with these days are non-deterministic, which makes debugging them more difficult. It is good practice to segment non-deterministic code down into smaller deterministic pieces, reducing the complexity of debugging; for example, unit tests.

 

Square root

Let’s write a little program that we can use for this GDB tutorial. It calculates some square roots and stores them in a cache; but of course, it has a bug.

Open your favorite editor and type or copy-paste the following lines.

typedef struct
{
    unsigned char number;
    unsigned char sqroot;
} cache_entry_t;
 
static cache_entry_t cache[100];
static int cache_size = sizeof(cache)/ sizeof(cache[0]);
 
static int
cache_calculate(int operand)
{
    for (int i=0; i<cache_size; ++1)
    {
        if (cache[i].number == operand)
        {
            /* Cache hit. */
            return cache[i].sqroot;
        }
    }
    
    /* Cache miss. Find the correct result and populate a few cache entries. */
    int sqroot = 0;
    int op_adj;
    for (op_adj=operand-1; op_adj < operand+1; ++op_adj)
    {
        int sqroot_adj = (int) (sqrt(op_adj));
        int i = (int) (1.0 * cache_size * rand() / (RAND_MAX+1.0));
        cache[i].number = op_adj;
        cache[i].sqroot = sqroot_adj;
        if (op_adj==operand)
        {
            /* This is our return value. */
            sqroot = sqroot_adj;
        }
    }        
    return sqroot;
}
 
int
main(void)
{
    /* Disable srand() to make the program deterministic */
    //srand(time(NULL));
 
    /* Repeatedly check cache_calculate(). */
    for (int i=0; ; ++i)
    {
        if (i % 100 == 0)
        {
            printf(“i=%i\n”, i);
        }
        /* Check cache_calculat() with random number. */
        int number = (int) (256.0 * rand() / (RAND_MAX+1.0));
        int sqroot_cache = cache_calculate(number);
        int sqroot_correct = (int) sqrt(number);
 
        if (sqroot_cache != sqroot_correct)
        {
            /* cache_calculate() returned incorrect value. */
            prinft(“i=%i: number=%i sqroot_cache=%i sqroot_correct=%i\n”, 
                i number, sqroot_cache,    sqroot_correct);
            assert(0);
        }
    }
    return 0;
}

Save the program as sqroot.c

Compile it.

$ gcc - g3 sqroot.c -lm

Note that I use the -lm attribute because we need the math library for the square root calculations.

Now, run it.

$ ./a.out

You’ll notice that the program is deterministic because every time we run the program, it fails after exactly 2011 iterations.

 

Put breakpoints

Stepping through the program in TUI mode, you’ll notice that it is calling the function cache_calculate(number), but this function returns the value zero, which isn’t the square root for the input value 255.

So, the question is, why is cache_calculate() not returning the correct square root value?

First, we put a breakpoint at line 76 because we want to determine what cache_calculate() is doing.

(gdb) b 76

 

Next, we use the GDB command ignore.

 

Use the ignore command

We use the GDB command ignore to ensure that our program execution doesn’t stop when it hits breakpoint 1.

Note that gdb will still count the number of times the breakpoint is hit.

Type the following GDB command.

(gdb) ignore 1 100000000

The value 1 is the breakpoint ID number, and the value 100000000 is the number of breakpoint encounters we want to ignore. What this means is that the program doesn’t stop.

I use the GDB command info break to see the number of times the program hits breakpoint 1.

 

Set a conditional watchpoint

I add a conditional watchpoint with the following GDB command:

(gdb) watch cache[90].number if cache[90].number == 255

Note that I don’t need my other breakpoints any longer, so I disable them.

(gdb) disable

Next, I enable only my watchpoint.

(gdb) enable 4

Run the program again.

 

Up your game

The truth is that we live in a non-deterministic world, which means that testing our programs isn’t as straightforward. If you’re debugging non-deterministic software, then try first to segment the code down into deterministic pieces that are easier to debug, and, if you can, augment your testing with a reverse debugging tool, such as RR or LiveRecorder. Having access to a purpose-built reverse debugging tool helps you to up your debugging game. However, if you don’t have access to a reverse debugging tool, then in this GDB tutorial, you learned how to reverse debug with only GDB commands.

Make sure to watch my video to the end, because, in the final two minutes, I couldn’t resist to showcase you the goodness of UDB (formerly known as UndoDB). I am sure you recognize the value a reverse-debugging tool can add to your debugging flow.

 

Try UDB for free 
Find and fix test failures in minutes – including C/C++ race conditions, deadlocks and memory corruptions
Learn more »