A flaky test passes and fails without a relevant code change. Repeated retries can conceal the signal and make a broken build look healthy, so first learn what changes between runs.
Collect a useful failure
- Record the test seed, worker count, browser, and relevant environment settings.
- Run the smallest failing test repeatedly, then compare isolated and parallel execution.
- Inspect shared files, global state, timers, network dependencies, and assumptions about test order.
Remove the nondeterministic dependency
Use unique test data, explicit setup and cleanup, deterministic clocks, and waits for observable state. Replace an external service with a controlled boundary when the test is meant to verify application logic.
Keep a retry only as a temporary containment measure with an owner and a deadline to remove it. Track the failure rate so a change can be shown to reduce flakiness rather than merely hide it.