Resources

Reverse debugging concurrency issues using multi-process correlation

MPC blog post image

The problem - concurrency

Companies requiring high performance have increasingly turned to various concurrent computing techniques in which several threads of execution operate simultaneously. This trend is largely a reaction to the fact that single core CPU speeds have plateaued since the 2000s.

Multithreading

One approach to achieving concurrency is multithreading. Multithreading allows the application to exploit several CPU cores simultaneously on a single machine. This comes at the expense of introducing several new potential failure modes (e.g. non-deterministic behaviour, read after write hazard, race conditions, etc.) which can make the application very complex and difficult to debug. Undo’s software flight recording technology helps developers reproduce, debug and diagnose these difficult types of defect.

Multi-process

To avoid the complexities of multithreading, some developers turned instead to making use of multiple processes on a single machine. Similarly to multi-threading, this technique allows the application to exploit several CPU cores simultaneously on a single machine. Additionally, it reduces the complexity and internal interdependencies within the application by relying on the operating system to enforce isolation between the processes’ unshared memory. 

Each individual process in a multi-process application is simpler and easier to debug in isolation than a monolithic multithreaded application. However, for processes that communicate with each other by manipulating data structures in a shared memory region, concurrency issues such as non-deterministic behavior, read after write hazards and race conditions still remain. Worse, this space is not well served by existing tooling. At Undo we wondered whether we could bridge the gap.

The challenge of recording and replaying multi-process

Undo’s Live Recorder already supports recording and replaying processes which access shared memory - irrespective of whether the other processes that write to the shared memory region are themselves being recorded. 

It does this by treating reads from shared memory like other non-deterministic inputs (such as file or network I/O) and storing the data in the recording file for injection at replay time. Live Recorder uses the system’s memory management unit (MMU) and JIT binary retranslation of the program to make this performant.

The advantage of this approach is that the recording file that Live Recorder creates is always self-contained. The recording can be replayed and debugged in isolation - even if the recorded process was part of a multiprocess application. 

When recording multiple processes in a multi-process application, this approach has the fundamental limitation that Live Recorder does not capture any information about the relative ordering of accesses to the shared memory region. This can make it difficult to reason about hazards and race conditions.

Multi-process Correlation for Shared Memory

We wanted to address this limitation with Live Recorder 5.0. The approach that we took was to build on the existing specialized support in Live Recorder 4.x for handling shared memory regions, and enhance it to store read and write events to the shared memory region in a new recording file which we call the “Shmem log”.

Shmem Log - MPC blog post

To achieve this, we create the Shmem Log as a memory-mapped file that is mapped into the address space of every process in the application. Each time a process touches the shared memory region, it also writes an event to the Shmem Log indicating that it has done so. Since the Shmem Log is a memory-mapped file, the operating system takes care of writing it out as a recording file.

In order to make this approach performant, we re-use the optimizations and JIT binary retranslation of the program from Live Recorder 4.x. We also make some assumptions - that all the processes within the application must be running under recording by Live Recorder and that the application creates its multiple processes by forking after the shared memory region is created.

This new approach means that we capture the ordering of the various processes’ accesses to the shared memory region, while retaining the ability to record and replay each process’ recording in isolation.

Now that we had an ordering of accesses to shared memory, we needed to provide a way to expose this information to a developer. We added a new ublame command to UndoDB (Live Recorder's interactive debugging tool used to replay the recordings generated by Live Recorder). The output looks like this:

Shmem Log

In this example, the ublame command tells the developer that two processes, with process IDs (PIDs) 18813 and 18809, have interacted with the data structure of interest in shared memory. 

We added an additional ugo inferior command to allow the developer to jump to a process and point in time of interest - in this example the point in time that a process last wrote to the data structure:

(udb) ugo inferior 187e23cf-aa8d   18,379:0x402666

Summary

With the ublame and ugo inferior commands developers can use Live Recorder 5.0 to diagnose multiprocess concurrency issues around data structures held in shared memory.

Future Work

For future Live Recorder releases, we are currently working on the following enhancements to the Shmem Log approach:

Consolidated recording file - The multiple recordings of separate processes each contain partial copies of the shared memory region, potentially containing duplicate information. We are investigating the pros and cons of generating a single consolidated and de-duplicated recording of the entire application. This would reduce recording file size, especially for long-running applications.

Multiple inferiors - The ugo inferior command in Live Recorder 5.0 allows the developer to jump to another process recording at a point in time of interest, but it does this by discarding the current process recording. 

IDE support - We’re investigating alternative ways to display the ordering information that is stored in the Shmem Log to the developer, including within the various IDEs that Undo support.

For more information

If you’d like to see a demo of MPC and other Live Recorder 5.0 features, watch this webcast.

Debugging concurrency defects