A Common Sense Guide to Symbols and Debug Info

This blog explains the difference between symbols and debug info, and suggests some ways to track down your “debug symbols”.

This isn’t a comprehensive guide to anything, and doesn’t talk about the innards of DWARF, ELF, binutils, GDB. It’s intended to be a friendlier guide than the many more rigorous guides.

Two essential resources for the absolute truth are:

https://sourceware.org/gdb/current/onlinedocs/gdb.html/Separate-Debug-Files.html

https://gcc.gnu.org/wiki/DebugFission

The information here isn’t specific to Undo’s tools, except for a couple of mentions and a section at the very end. The impact on Undo’s tools is nearly identical to live debugging or loading a core file, so although you see UDB in the screenshots, this is just as relevant for plain GDB.

Who is this for?

Anyone who has encountered this:

More specifically, you’ll get this when you’re trying to debug a stripped binary: one without debug info or symbols. You’re most likely to encounter this when you’ve built on one machine then run the binary on another. Four cases where this may happen are:

• Normal interactive debugging on a remote host
• Debugging a core file generated somewhere else
• Using gdbserver on a remote host
• Debugging an Undo recording

What are “symbols”? What is “debug info”? What are “debug symbols”?

Put simply, symbols are the names and addresses of functions and variables in your program. Debug info is all the extra information needed to tie your machine code to your source code. For example, line information (what machine code addresses correspond to this source line) or local variable info (what register is this variable in when we are at this address?).

Symbols are included in executables by default. Debug info is not, and you need to pass -g to the compiler to get it.

If you try to debug a fully stripped program, your backtraces will look like this:

Most stack frames show only a disassembly address, with the occasional symbol name scattered in.

If you only have symbols, it’s better, but it’s still not going to give you a good experience. You can see what function you’re in and set some more useful breakpoints, but source code debugging is unavailable:

If you have symbols and debug info, you’re in the happy place:

I don’t have a strict definition for the phrase “debug symbols”, as it’s not a strictly defined concept. It seems to refer to “everything you need for debugging” – that being both symbols and debug info. You can’t actually have debug info without symbols, so it’s a slightly odd phrase. I suspect people use it because for a developer hoping to debug a program, the difference between symbols and debug info is pretty irrelevant. You really need both. Unless you can debug compiled machine code as easily as source code… in which case you have achieved oneness with the computer and you don’t need to be here. Well done.

Why is this happening to me?

To answer this, let’s first say what has happened. The debug information and/or symbols for your program have been removed (or stripped) from the binary on the system you’re trying to debug on. This can happen at compile time or as a post-processing step after compilation.

There are some very good reasons people choose to do this:

1. Size
2. Secrecy

On size, consider the gdb binary itself:

If you add this size to the sizes of any shared objects, you can end up with a very large project, and you almost can guarantee that an end user somewhere will have a slow enough data transfer to be annoyed by all this useless information.

Note that I’m only talking about disk space here. Neither symbols nor debug info are loaded into memory when the program executes. They’re not needed at run time so would be a waste of RAM. An exception to this is dynamic symbols in the .dynsym and .dymstr sections. These are needed at runtime by the dynamic linker. This is why some symbol names still appear (and are breakpointable) in a fully stripped binary.

Symbol names take a lot less space, but they do give away information about how your program works. It’s much more difficult to decipher any meaning from a large amount of totally context-free machine code.

Some vendors (including Undo) choose to include symbols and debug information in their shipped binaries. In our case the debug info is relatively small and nothing valuable can be inferred from symbol names or filenames. More importantly, we find being able to debug our code on a remote site to be very useful. The cost-benefit analysis is in favor of shipping everything.

Where are my symbols?

Ideally this is a question to ask your build system team. They may have a really great answer (more in a moment). However, this isn’t always possible. Maybe they don’t respond quickly, or you don’t want to bother them, or they have other priorities at the moment. I’ve had to sniff out debug information from a fair number of customer environments, so I’ll share how I do it.

Note: It’s worth knowing that debug information is in a format called DWARF, which is composed largely of offsets. Everything is defined as an offset from everything else. For this reason you need the *exact* debug info for your binary. Anything “close” just won’t work, at all, and will waste your time.

It’s also possible that you don’t have any symbols or debug info anywhere in your build. It’s usually somewhere, but this is another good way to waste time.

Caveats aside, if you’re sure you’ve got the same build that your debugged process came from, let’s have a look.

The first step is to see if there’s an unstripped version of the binary in the build tree. Use find <build tree> -name "mybinary", and then file to check.

If you see this output:

$ file mybinary_unstripped

mybinary_unstripped: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=1d904bc819c5b7030ec0e2b54ee6034acbea4194, for GNU/Linux 3.2.0, with debug_info, not stripped

This is ideal and can be used for debugging.

If your program doesn’t include any shared objects, you’re done. If it does, you need to find the debug symbols for them too if you want to debug them. I suggest finding the debug info for a couple of shared objects: that should give you enough information about the structure of the build. I’ll explain what to do with this in a moment.

If you see this output:

$ file mybinary_strippeddebug

mybinary_strippeddebug: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=1d904bc819c5b7030ec0e2b54ee6034acbea4194, for GNU/Linux 3.2.0, not stripped

$ file mybinary_fullystripped

mybinary_fullystripped: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=1d904bc819c5b7030ec0e2b54ee6034acbea4194, for GNU/Linux 3.2.0, stripped

These mean you still have hunting to do.

Try searching the project for *.debug, *.dwo, *.dwp files. These are various different ways that separate debug files can be stored. In many cases, there will be a link embedded in the stripped binaries (either the .gnu_debuglink section or the .debug_str section for DWO) which gdb will be able to follow and load the corresponding symbol file.

If you haven’t found anything yet, it’s possible your project isn’t built with debug info, in which case you’re back to speaking to your build team or investigating your build infrastructure code. Look for “strip”, “objcopy” and “split-dwarf”.

If you’ve managed to find some debug info, great! The next step is to tell GDB about it.

Loading the symbols

GDB has many commands for loading symbol files once you’ve located them, for example set debug-file-directory, set solib-search-path, symbol-file, add-symbol-fileetc.. Unfortunately, each command is designed for a different scenario, and it’s not obvious which command applies in which scenario.

Rather than going through the scenario each command goes with, I will introduce the multi-tool of symbol loading: add-symbol-file.

This isn’t always the simplest way, but it is the most reliable. It can cope with everything from a simple executable to more complex and surprising situations.

The command works like this:

add-symbol-file symbolfile -o offset

You get the offset from the memory map of the process. In GDB, use info proc mappings – you need the lowest memory address of the library or executable you’re debugging. For example, if you are loading the symbols for libcool.so in this output:

Supposing your debug symbols were at /home/work/ismith/project/build/dbg/libcool. so you would use:

add-symbol-file /home/work/ismith/project/build/dbg/libcool.so -o 0x7ffff7f29000

Can’t I just debug the unstripped binary?

Yes. If this is an option for you, you can absolutely debug the unstripped binary. If you want to debug on-target you’ll need the source code as well. This may not be an option if:

• There isn’t enough disk space on a small target device
• You’re not allowed to put symbols/source there – e.g. don’t copy your secret project to AWS

But if it is an option, go for it. As previously mentioned, debug info is neither required nor used in any way by a running process. It will not affect its behavior – on the other hand, attaching a debugger to it will massively affect its timing behavior which may cause problems. This isn’t due to the debug info though.

I looked for the debug builds and I found dbg, rel, dbgrel, dbgopt, reldbg, relwithdbginfo, which one do I use?

It’s common to have a lot of different build targets in a project. Many build systems provide several targets by default and your project may have built on these. This is because there are some orthogonal choices when making a build:

1. Build with or without debug info (-g options)
2. What optimization level to use (-O options)
3. Project-specifc “debug” modes by defining DEBUG or similar (-D compiler options). Might enable extra checking or extended error messages for developers.

If you have lots of build targets then you probably have a partial (or even full) cross product of these options. For best debugging, you want low optimization and debug info, and probably with as many -DDEBUG type options as you can. With high optimization levels, some source code lines have no corresponding machine code, and others will have machine code reordered to be most efficient. So when debugging, you’ll encounter optimized out variables (https://undo.io/resources/value-optimized-out-reverse-debugging-rescue/) and experience the current location jumping around when you step.

Often “debug builds”, with 2 and 3, are too slow to be used for anything large, so you’ll have to debug optimized code. This is painful to begin with but there are certain tricks and reflexes you’ll learn that make it pretty easy. Then debugging low optimization code becomes a special treat.

If you’re using UDB, the ugo undo or uu command is very useful, as it takes you back to your previous place. So if a next took you too far, you can quickly go back and try a step instead, or even stepi through the assembly code.

Symbol files and Undo

The issue of locating and loading separate debug symbols is a subject close to our hearts at Undo. Our technology records processes in a minimally invasive way – this means we can, for example, record a critical process on a network switch without interfering with its operation. In such a scenario we almost always record a release build with separate debug symbols. Since the replay part of the technology is a debugger (UDB) we absolutely need to find those symbols and get them loaded.

On the other hand, if the symbols are available on the device, Undo (LiveRecorder or UDB) will find them and add them to the recording. At replay time, Undo constructs a sysroot inside /tmp and uses set sysroot to point to it.

All of our customers have complex environments, and many have to deal with separate symbols/debug info. We offer a “Debug Info Service” which can serve symbols via debuginfod, or we can work with you to use existing scripts or mechanisms with Undo for a one-step load and debug experience.