Multiprocess debugging in GDB

Intro

GDB lets you debug and inspect multiple processes at once. In this tutorial we’ll inspect the operation of the system bash; that is, we can see a lot of what is going on even when we don’t have any debug info or source code.

Catching the Write Syscall

We can create two bash sessions, echo $$ to get the Process ID in one and in the other open a GDB session on that PID.

$ echo $$
1633190

$ gdb -p 1633190
(gdb) info proc
process 1633190
cmdline = ‘bash’
cwd = ‘/home/user/’
exe = ‘/usr/bin/bash’

Now the first terminal will appear frozen, as it is now paused for inspection by GDB in the other window. i.e. if you try to use the first terminal, your actions won’t appear until you continue in GDB. In GDB we can set catchpoints to automatically pause the process at a specific time during execution, rather than having to guess where to put breakpoints. To start with, we’re going to catch the “write” syscall.

(gdb) catch syscall write
Catchpoint 1 (syscall ‘write’ [1])

When we type, bash echoes every character. Echoing uses the write syscall, and so the GDB session pauses the first terminal as soon as you type a character in the 2nd terminal – or rather, when the bash process enters the write syscall. From here you can view the backtrace and disassemble the process.

(gdb) bt
# view backtrace
(gdb) disas
# view dump of assembler code for current function

Continuing from here will cause the character you typed to appear in the first terminal.

Catching Fork and Exec

As it is not very useful to catch every use of the write syscall, we’ll remove that catchpoint and instead catch “fork” and “exec”.

gdb) d 1           # delete catchpoint 1
(gdb) catch fork
Catchpoint 2 (fork)
(gdb) catch exec
Catchpoint 3 (exec)

Catching “fork” instead of “syscall fork” allows it to catch any type of fork, such as clone and clone2, instead of just the base “fork” syscall. Similarly we will catch “exec” not “syscall exec” to catch every flavour of exec. We’ll now try running something in our first bash terminal again – this time we’ll cat a file.

$ cat /etc/issue

After executing this, the 2nd terminal will pause again and GDB shows we have stopped at a fork in the bash process. If we look at the inferiors however, we can see there is still only one process. This is because GDB has stopped before the fork has finished executing. What usually happens here is that GDB will detach from the child process and let it run. But we want to debug both the bash and the cat process at once; happily we can adjust what GDB does here:

gdb) info inferiors
  Num  Description         Connection       Executable
* 1    process 1633190     1 (native)       /usr/bin/bash
(gdb) set detach-on-fork off
(gdb) nexti                # Allow the fork to execute
[New inferior 2 (process 1639264)]
(gdb) print $rax
£3 = 1639264
(gdb) info inferiors
  Num  Description         Connection       Executable
* 1    process 1633190     1 (native)       /usr/bin/bash
  2    process 1639264     1 (native)       /usr/bin/bash

So now we are debugging two processes at the same time. This is right after the fork, so they’re both bash, but they’re different processes with different PIDs.

And now inspecting the backtrace you can see it GDB is now in the wait system call, having created the cat process and waiting for it to complete. If we switch to inferior 2, which is the child process, and continue it, we’ll hit the exec catchpoint as it executes “cat”.

(gdb) inferior 2
[Switching to inferior 2 [process 1639264] (/usr/bin/bash)]
(gdb) continue
Continuing.
process 1639264 is executing new program: /usr/bin/cat
…
(gdb) info inferiors
  Num  Description         Connection       Executable
  1    process 1633190     1 (native)       /usr/bin/bash
* 2    process 1639264     1 (native)       /usr/bin/cat

We can now continue the inferior 2 process, and this will allow the cat process to execute in our first bash terminal.

(gdb) continue
[Inferior 2 (process 1639264) exited normally]

$ cat /etc/issue
Ubuntu 22.04.1 LTS \n \l

As the parent process hasn’t completed however our 2nd terminal is still stuck and cannot be interacted with. As inferior 2 has stopped running, we can swap back to inferior 1 and continue it, and the wait will complete as the child has exited.

(gdb) inferior 1
[Switching to inferior 1 [process 1633190] (/usr/bin/bash)]
(gdb) continue
Continuing.

And now looking back at the first terminal the process has entirely completed and you can execute your next command.

Conclusion

This has been a fairly brief example of catching syscalls and pseudo-syscalls in fork and exec using catchpoints, as well as getting fairly interesting information when debugging even without debuginfo or source code. We’ve also shown how you can debug multiple inferiors using the same GDB.

GDB Training 
Master GDB and save time debugging complex codebases. For teams of C++ engineers wanting to get more productive when debugging.
Learn more »

Don’t miss the next GDB tutorial: sign up to the gdbWatchPoint mailing list below.