Apart from gdb and lldb, rr is another powerful debugger that is quite helpful in debugging multi-threaded code which is harder to debug due to its non-determinism. Often, when debugging multi-threaded code, a certain section of code triggers a bug but fails to reproduce in subsequent executions of the program.

The rr debugger can perform a reproducible debugging session for non-deterministic, multi-threaded code. It does this by recording the debugging session, which you can replay and track incrementally to close in on the issue. It does this by first saving the trace of the program to disk with all the required information to reproduce the program execution.

One of the limitations of rr is that it is only supported on Linux and on Intel CPUs as of now. To set up the rr debugger on Ubuntu 16.04, we'll pull the latest .deb package from https://github.com/mozilla/rr/releases. At the time of writing, the rr version is at 5.2.0. Having downloaded the deb package, we can then install rr by running the following code:

sudo dpkg -i https://github.com/mozilla/rr/releases/download/5.2.0/rr-5.2.0-Linux-x86_64.deb

Note: It's a prerequisite to install perf tools. You can install them as follows:

sudo apt-get install linux-tools-4.15.0-43-generic

Replace linux-tools-(version) as applicable for your kernel version. You can get the kernel version with the uname -a command on Linux. Another prerequisite is to set the sysctl flag in perf_event_paranoid from 3 to -1. It's recommended that you set this temporarily by running the following code:

sudo sysctl -w kernel.perf_event_paranoid=-1

With that done, let's quickly create a new project by running cargo new rr_demo and go through a debugging session with rr. We'll explore how to use the rr debugger for a sample program which demonstrates non-determinism. We'll depend on the rand crate, which we can add to the Cargo.toml file by running cargo add rand. We have the following code in our main.rs file:

// rr_demo/src/main.rs

use rand::prelude::*;

use std::thread;
use std::time::Duration;

fn main() {
    for i in 0..10 {
        thread::spawn(move || {
            let mut rng = rand::thread_rng();
            let y: f64 = rng.gen();
            let a: u64 = rand::thread_rng().gen_range(0, 100); 
            thread::sleep(Duration::from_millis(a));
            print!("{} ", i);
        });
    }
    thread::sleep(Duration::from_millis(1000));
    println!("Hello, world!");
}

This is a minimal, non-deterministic program that spawns 10 threads and prints them to stdout. To highlight the reproducibility aspect of rr, we'll spawn threads and sleep for a random duration.

First, we need to record the program's execution with rr. This is done by running the following code:

rr record target/debug/rr_demo

This gives us the following output:

On my machine, this records and stores the program execution trace at the following location:

rr: Saving execution to trace directory `/home/creativcoder/.local/share/rr/rr_demo-15'

The recorded file, rr_demo-15, might be named differently on your machine. We can now replay the recorded program by running the following code:

rr replay -d rust-gdb /home/creativcoder/.local/share/rr/rr_demo-15

The following is the session with gdb running under rr:

As you can see, the same sequence of numbers gets printed every time, as the program is being run from the recorded session in the previous run. This helps to debug a multi-threaded program where threads run out of order and might not reproduce the bug when you run your program next time.