Build goes red. You look at it for about four seconds, decide it's probably nothing, and hit rerun.
It goes green. You move on.
I do this. Most people I work with do this. And the uncomfortable thing is that it's a completely rational habit, because most of the time you're right.
The numbers are genuinely bad
I went looking for how common this is, and the public data is worse than I assumed.
Atlassian's engineering team found flaky tests caused 21% of their master branch build failures, costing an estimated 150,000 developer hours a year. Google reported that around 16% of their tests flaked at least once in a 30-day window, and that 84% of pass-to-fail transitions came from flakiness rather than actual bugs.
Slack published something even starker. Their mobile test failures were running at 56.76% flaky before they built detection and suppression for it. They got it down to 3.85% and saved roughly 553 hours of triage.
GitLab keeps their maths in the open, which I appreciate. Their handbook works out that flaky tests waste 31,395 CI minutes a month across roughly 91,000 pipelines, which is about $2,653 in compute alone, before anybody's time.
So when I hit rerun assuming it's flaky, I'm right most of the time. That's exactly what makes it dangerous.
Three different failures that look identical
Here's the actual problem, and it's not really about flakiness.
A red build tells you something failed. It doesn't tell you which of three completely different things happened, and they need three different people to fix them.
The code is broken. Your change genuinely doesn't work. Fix the code.
The test is broken. A race condition in the test, a hardcoded sleep that works on a fast runner and not a slow one, shared state left over from another test. Fix the test.
The infrastructure is broken. The agent is missing a tool, a dependency wouldn't download, the runner ran out of disk. Fix the environment.
All three produce a red X. From the build list, they're indistinguishable. And the rerun button only helps with two of them, which is enough to teach you the wrong lesson.
Sauce Labs makes a point about this that I keep thinking about: retry logic masks real problems. A test failing intermittently because of a genuine race condition in your application code will pass on retry. The pipeline goes green. The race condition ships.
The rerun didn't fix anything. It just stopped you looking.
What actually separates them
Coming from electronics, this feels familiar. An intermittent fault on a board is the worst kind, because the instinct is to power cycle it and see if the problem goes away. It usually does. That tells you nothing about whether the fault is still there.
What you need is more than one reading, and you need them from different places at the same moment.
Same thing here. To tell those three failures apart, you need three things together:
What the build actually output, not the summary line but the console around where it failed.
What changed in the code since the last green run.
Whether other builds on the same agent are also failing.
That third one is the tell for infrastructure problems and it's the one people check last, because it means going and looking at other jobs that have nothing to do with your change.
How I do it now
This is where I use Vörr, and it's mostly about not having to go to three places.
I pull the recent deploy and build events across the pipelines, so I can see whether this job alone is failing or whether several are. If three unrelated jobs went red in the same twenty minutes, that's almost never three people's code.
Then I pull the logs for the failing job and read the actual output, not the status.
Then, if it looks like it might be the code, I look at what changed. Vörr reads the repositories too, so I can see recent commits on that repo and trace the function involved without opening GitHub in another tab.
Three questions, one place. The whole point is that checking properly now costs about as much as hitting rerun, which is the only reason I actually do it.
The five patterns worth knowing
Most flakiness clusters into a small number of causes, and knowing them makes the console output much faster to read.
Async operations asserted before they finish. Reportedly the single most common one.
Hardcoded waits. A sleep(2000) that passes on a fast runner and fails on a slow one.
Shared state between tests. Passes alone, fails in the suite, or fails only when the order changes.
Test pollution from data that wasn't cleaned up.
Real race conditions in the application, which is the dangerous category, because this is a genuine bug wearing a flaky test's clothes.
If you can place a failure into one of those five in thirty seconds, you've already done more than a rerun would.
What I'd actually change
Log the reruns. If your CI doesn't record how many times a job was retried before it went green, you have no idea what your flake rate is. GitLab tracks average retry count per pipeline, which is a simple number and tells you a lot.
Read the console before you rerun, even for ten seconds. unknown command, no space left on device, connection refused are infrastructure. A test assertion is not. That's a two-second read that changes who should be looking at it.
Quarantine rather than rerun, for tests you know are unreliable. A test that's always retried is a test nobody trusts, and it should be out of the critical path and on somebody's list rather than silently eating minutes.
And check whether other jobs failed at the same time before assuming it's yours. Costs nothing, and it's the fastest way to find out that the problem isn't your code at all.
The rerun button is fine. It's the not-looking that costs you.
Author note
Manjunaathaa, Associate DevOps Development Engineer at Frigga Cloud Labs.
I came to infrastructure from electronics, and intermittent faults were the thing I was warned about most. A board that fails once in twenty runs is much harder than one that fails every time, because the failing case is where all the information is and you keep throwing it away.
Red builds are the same and we treat them the opposite way. The rerun discards the one run that had something to tell you.
Still working out a good habit for this. If your team has one that actually sticks, I'd like to hear it. LinkedIn.