TEMPLATE: Header + Content + Left and Right Sidebar

If you ever used a debugger, I bet you know how frequent, and how annoying “optimized out” messages are when you are looking for values of your variables.

(udb) print a
$1 = <optimized out>

This message is of course a legitimate outcome of compilers making our code efficient at runtime, and even in a perfect world, we are not going to get rid of this.

But what if you are debugging an issue right now, and desperately need to see the value? There are two well-known options, both with huge downsides:

• rebuild the code with optimizations disabled
• try to recover the value by manual inspection of registers and memory contents

If you go for a rebuild, you throw away your current reproduced case (a core dump or a live process), spend time rebuilding the code (which can be a long open-ended journey), and then hope that your issue reproduces with debug-enabled binaries (some bugs become just not reproducible this way).

If you try to recover the value manually, you need to be fluent in assembly-level debugging, and then you just have a tedious and error-prone job to do, with chances of just not being able to gather enough data.

Fortunately, today there’s a better option – process recording and reverse debugging.

Just keep reversing

In fact, even in an optimized build, you can see such values, but just for small amount of execution steps until these values are displaced from registers or memory by other values handled by the program later on. You can see this if you execute your program step by step, starting with the time when the variable is set, until some later time when its value becomes optimized out.

Naturally, when you step in the reverse direction, you can see how your optimized out value becomes reachable!

Of course you don’t have to go backwards to reach your goal, but it is much more efficient in debugging practice.

With UDB (Undo’s time travel debugger), you can examine the state of your process at any moment in time – at a granularity of single machine instruction. This, in turn, brings a paradigm shift in troubleshooting workflow: the reverse debugging.

How does it work?

Binaries of executable formats contain special sections of information to facilitate debugging.

One bit of such information is a table which specifies the expression which the debugger can use to obtain the value of a variable. Each such expression is valid only within a defined range of instruction pointer (aka program counter) values.

But let’s get specific. We will review a simple example program compiled into a Linux executable.

Linux executables are in ELF file format.

The debugging data in ELF files follows DWARF format.

Let’s dump some debug information from our test program and look for description of a variable with such a command:

readelf --debug-dump=info example

The interesting part is

<2><9c>: Abbrev Number: 5 (DW_TAG_variable)
<9d> DW_AT_name : a
<9f> DW_AT_decl_file : 1
<a0> DW_AT_decl_line : 3
<a1> DW_AT_decl_column : 9
<a2> DW_AT_type : <0x34>
<a6> DW_AT_location : 0x4 (location list)
<aa> DW_AT_GNU_locviews: 0x0

So the location information for a variable is at the offset 0x4 in the list.

Let’s dump location information:

readelf --debug-dump=loc example

Contents of the .debug_loc section:

Offset   Begin            End              Expression

…

00000004 v000000000000000 v000000000000000 views at 00000000 for:
0000000000000518 000000000000051c (DW_OP_reg0 (rax))

This tells us that we can get the value when instruction pointer (also known as Program Counter) has offset between 0x518 and 0x51c, and that the value resides in the rax register.

Let’s get a fine grasp of this by inspecting actual values in an interactive debugging session. This can be done either with a recording or with a live debuggee process.

(udb) break main
Breakpoint 1 at 0x55dc15dc4510: file example.c, line 4.
(udb) continue
Continuing.

Breakpoint 1, main () at example.c:4
4           a = rand();

We will use display commands here to ask debugger to show the values of the variable, and of the Program Counter register, whenever it gives us a prompt.

(udb) display a
1: a = <optimized out>
(udb) display $pc
2: $pc = (void (*)()) 0x55dc15dc4510 <main>

Remember that debugger stops before executing the line of code it shows you. So it makes sense that value of a is unknown before it is assigned. This also agrees with the information from DWARF data – PC value ends with 0x510, and value will be known only after PC value reaches 0x...518.

(udb) next
6           b = rand();
1: a = 1804289383
2: $pc = (void (*)()) 0x55dc15dc4518 <main+8>

After the first next command, which advances us to the next line of code, a is already initialized.

(udb) next
7           b = total += b;
1: a = 1804289383
2: $pc = (void (*)()) 0x55dc15dc451d <main+13>

Value of a is still known. PC register is shown to have 0x...51d value, but that’s the address of the instruction it is about to execute. In other words, the instruction at address 0x...51d hasn’t been executed yet, and the program state corresponds to a lower PC value.

(udb) next
8           b = c = rand();
1: a = <optimized out>
2: $pc = (void (*)()) 0x55dc15dc451f <main+15>
(udb)

Another next, and a is gone. This agrees with DWARF data – PC value is now higher than the upper bound provided.

Want to try reverse debugging on your own code? Get the free trial of UDB.