Debugging flaky tests in Intellij IDEA

If an old unit test suddenly fails on your machine and you forget to do something about it, it can be a long time before you meet again.

Probably the name of the test contains exciting words like Concurrent, Sleep or Async. Probably the author of the code resigned a couple of years ago. Anyway, you want to be sure that this test won’t fail again at the worst possible time.

I’ll show a couple of IntelliJ IDEA features which usually help me to catch and fix such stuff.

Retry test until failure

I couldn’t think of a good made-up example, so let’s just write a crappy test which fails in about 5% of cases:

package com.example;

import org.junit.Test;
import java.util.concurrent.ThreadLocalRandom;
import static org.junit.Assert.assertTrue;

public class UnstableTest {
    @Test
    public void willFailOccasionally() {
        double value = ThreadLocalRandom.current().nextDouble();
        assertTrue(value > 0.05);
    }
}

If we run it once, there’s a good chance that it will pass.

Let’s edit a Run Configuration for this test (Alt+Shift+F10).

Set “Repeat” to “Until Failure”:

Launch the test again.

Now we have a proof that the test really fails. We can also inspect the data during the failing iteration if necessary.

Set breakpoint on specific exception

Open the Breakpoints window (Ctrl+Shift+F8) and add new Java Exception Breakpoint. According to the previous screenshot, we need a breakpoint on AssertionError. No additional settings are necessary, just make sure that our custom breakpoint is checked on the left panel.

After this, running the test in debug mode will make it stop on the exception. Now we can jump to any stack frame and inspect the data.