Detect C/C++ data races faster with advanced fuzz testing

Finding rare race conditions in multithreaded applications can be quite a challenge. So we introduced Thread Fuzzing in Undo. Thread Fuzzing is a feature which perturbs the regular scheduling of threads, in order to increase the frequency of concurrency bugs. It’s a way of shaking the codebase to see what comes out, letting you find race conditions before they make it into production. For more information on Thread Fuzzing, check out the docs.

Introducing Feedback-directed Thread Fuzzing

In our latest release, Undo Suite 8.0, we took this a step further and introduced Feedback-directed Thread Fuzzing.

Feedback-directed Thread Fuzzing enables you to detect more of these bugs faster. We’ve seen it hit bugs in seconds that normally take 30 minutes. One engineer from a leading networking company confirmed this new capability has become essential in their debugging process.

How Feedback-directed Thread Fuzzing works

Feedback-directed Thread Fuzzing adds an analysis step before the actual fuzzing.

To do this, you first need to make a reference recording. This is just a normal Undo recording – it doesn’t need to be long, and it doesn’t need to have reproduced the bug. It just needs to have run the multithreaded code that’s under suspicion. Undo then analyzes that recording to find the data that’s accessed by multiple threads, and the instructions that access those areas.

The new live-record --thread-fuzzing-analyze option examines an Undo recording to identify instructions that access memory shared between multiple threads. This analysis step finds the code that’s accessing shared data, possibly incorrectly.

The second step then increases the rate of thread switches around those instructions when re-recording, helping expose hidden concurrency bugs.

See Feedback-directed Thread Fuzzing in action

As you can see from the product tour above, Feedback-directed Thread Fuzzing operates in multiple passes:

• An analysis step: an existing recording of your program is analyzed to find data that’s accessed by multiple threads and the code that accesses that data. Note that this recording doesn’t need to exhibit any failures.
• A fuzzing step: the target instructions found in the analysis step are used to select when to switch threads. This can be run multiple times with the same analysis results. Each time the fuzzing step is started a different random seed is used.

A software engineer working on networking software explains in this article how he’s used this new feature for root-cause analysis and found that the cause of the crash he was investigating was heap-internal book-keeping data being overwritten by a memcpy.