If you ever used a debugger, I bet you know how frequent, and how annoying "optimized out" messages are when you are looking for values of your variables.
(udb) print a
$1 = <optimized out>
This message is of course a legitimate outcome of compilers making our code efficient at runtime, and even in a perfect world, we are not going to get rid of this.
But what if you are debugging an issue right now, and desperately need to see the value? There are two well-known options, both with huge downsides:
- rebuild the code with optimizations disabled;
- try to recover the value by manual inspection of registers and memory contents.
If you go for a rebuild, you throw away your current reproduced case (a core dump or a live process), spend time rebuilding the code (which can be a long open-ended journey), and then hope that your issue reproduces with debug-enabled binaries (some bugs become just not reproducible this way).
If you try to recover the value manually, you need to be fluent in assembly-level debugging, and then you just have a tedious and error-prone job to do, with chances of just not being able to gather enough data.
Fortunately, today there's a better option - process recording and reverse debugging!
Just keep reversing
In fact, even in an optimized build, you can see such values, but just for small amount of execution steps until these values are displaced from registers or memory by other values handled by the program later on. You can see this if you execute your program step by step, starting with the time when the variable is set, until some later time when its value becomes optimized out.
Naturally, when you step in the reverse direction, you can see how your optimized out value becomes reachable!
Of course you don't have to go backwards to reach your goal, but it is much more efficient in debugging practice.
With Undo's software failure replay platform, you can examine the state of your process at any moment in time - at a granularity of single machine instruction. This, in turn, brings a paradigm shift in troubleshooting workflow: the reverse debugging.
How does it work?
Binaries of executable formats contain special sections of information to facilitate debugging.
One bit of such information is a table which specifies the expression which the debugger can use to obtain the value of a variable. Each such expression is valid only within a defined range of instruction pointer (aka program counter) values.
But let's get specific. We will review a simple example program compiled into a Linux executable.
Linux executables are in ELF file format.
The debugging data in ELF files follows DWARF format.
Let's dump some debug information from our test program and look for description of
a variable with such a command:
readelf --debug-dump=info example
The interesting part is
<2><9c>: Abbrev Number: 5 (DW_TAG_variable)
<9d> DW_AT_name : a
<9f> DW_AT_decl_file : 1
<a0> DW_AT_decl_line : 3
<a1> DW_AT_decl_column : 9
<a2> DW_AT_type : <0x34>
<a6> DW_AT_location : 0x4 (location list)
<aa> DW_AT_GNU_locviews: 0x0
So the location information for a variable is at the offset
0x4 in the list.
Let's dump location information:
readelf --debug-dump=loc example
Contents of the .debug_loc section:
Offset Begin End Expression
00000004 v000000000000000 v000000000000000 views at 00000000 for:
0000000000000518 000000000000051c (DW_OP_reg0 (rax))
This tells us that we can get the value when instruction pointer (also known as Program Counter) has offset between
0x51c, and that the value resides in the
Let's get a fine grasp of this by inspecting actual values in an interactive debugging session. This can be done either with a recording or with a live debuggee process.
(udb) break main
Breakpoint 1 at 0x55dc15dc4510: file example.c, line 4.
Breakpoint 1, main () at example.c:4
4 a = rand();
We will use
display commands here to ask debugger to show the values of the variable, and of the Program Counter register, whenever it gives us a prompt.
(udb) display a
1: a = <optimized out>
(udb) display $pc
2: $pc = (void (*)()) 0x55dc15dc4510 <main>
Remember that debugger stops before executing the line of code it shows you. So it makes sense that value of
a is unknown before it is assigned. This also agrees with the information from DWARF data - PC value ends with
0x510, and value will be known only after PC value reaches
6 b = rand();
1: a = 1804289383
2: $pc = (void (*)()) 0x55dc15dc4518 <main+8>
After the first
next command, which advances us to the next line of code,
a is already initialized.
7 b = total += b;
1: a = 1804289383
2: $pc = (void (*)()) 0x55dc15dc451d <main+13>
a is still known. PC register is shown to have
0x...51d value, but that's the address of the instruction it is about to execute. In other words, the instruction at address
0x...51d hasn't been executed yet, and the program state corresponds to a lower PC value.
8 b = c = rand();
1: a = <optimized out>
2: $pc = (void (*)()) 0x55dc15dc451f <main+15>
a is gone. This agrees with DWARF data - PC value is now higher than the upper bound provided.
Author: Andrey Utkin, Software Engineer at Undo