Flaky Tests & Reliability

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

The reference material consulted while preparing these pages is listed in the section bibliography.

A flaky test is one that passes and fails on the same code and the same environment. Flakiness is the main cost of a browser test suite: once a team learns that a red build is "probably the flaky one", real regressions are merged unnoticed. This page explains the causes, shows how Playwright, Cypress and Selenium each defend against them, and then works one example from end to end: a deliberately racy test against the Bookshelf demo application, reproduced on demand, classified by retries, quarantined, diagnosed and fixed. It targets Playwright 1.64, Cypress 16.2 and Selenium 4.51 and was checked on 25 October 2026. The examples ran against Bookshelf (described in Example Application & Test Layout) in headless Chromium, or in Electron for Cypress, on macOS. Cypress Cloud and Test Replay need an account and were not run; their sections say so.

The tool-specific mechanics are covered where they belong, and this page links to them instead of repeating them: Playwright: Parallelism, Sharding & Retries, Playwright: Locators, Actions & Auto-Waiting, Cypress: Core Concepts, Cypress Cloud and Selenium: Waits & Expected Conditions. How retries and sharding fit a pipeline is in Running in CI/CD.

The experiment used on this page

A flake that never happens cannot be demonstrated, so the example has to fail by design. Bookshelf saves a review with POST /api/reviews, and its page shows the new review at once as a pending row (data-state="pending") that becomes saved when the server answers. The server delay is normally constant. For this page a copy of the Bookshelf server got one extra line that adds a random delay of 0 to 300 ms on top of BOOKSHELF_REVIEW_DELAY_MS, switched on with BOOKSHELF_REVIEW_JITTER_MS. Everything else is the application described in the example-application page.

# A copy of the Bookshelf server whose review save takes a random 0-300 ms
PORT=3966 BOOKSHELF_REVIEW_DELAY_MS=0 BOOKSHELF_REVIEW_JITTER_MS=300 node server.js

Source: Node.js: Command-line API

The random delay stands in for what makes real tests flaky: a loaded CI machine, a slow database, a cold cache, a busy third-party service. The test submits a review as the signed-in reader and checks that the row was saved. Two versions of it exist in every tool: one that guesses how long the save takes, and one that waits for the condition.

Why tests are flaky

Almost every flake is a test that assumes something about timing, state or the environment that is not guaranteed.

Cause What happens Typical remedy

Timing and races

The test acts or asserts before the application reaches the state it expects: a fixed sleep that is too short, a one-shot read of the DOM, a click on an element that is still being replaced. Passes on a fast machine, fails under load.

Wait for a condition, never for time. Use auto-waiting actions and retrying assertions.

Shared state

Tests share a database, a user, a cookie or a counter. A test that assumes "there are three reviews" fails when another worker adds one. Parallel runs expose this immediately.

Own your data: unique names, per-test users, API seeding, a fresh browser context per test.

Test order

A test passes only because an earlier test left the data it needs, or fails because an earlier test left data behind.

Make tests independent; run them in random order and in isolation.

Animations and transitions

The element moves, fades or is covered by an overlay while the test clicks it.

Rely on actionability checks; disable animations for tests (CSS, prefers-reduced-motion, or the tool’s option).

Network

A slow or failing API, an external script or an analytics call changes how long a page needs. A third-party service is unavailable.

Stub or mock external services; wait for the specific response when it matters.

Time and randomness

Dates, clocks, random identifiers, sorting ties and locale-dependent formats change between runs.

Control the clock, seed random values, assert on stable text.

Environment and load

Different browser versions, headless versus headed, screen size, fonts, an undersized CI runner or too many parallel workers starve the browser and stretch every step.

Pin versions, run the same image locally and in CI, size workers to the machine.

Real defects

An intermittent bug in the application: a race in the product, not in the test.

A flaky test is sometimes right. Investigate before you add a retry.

Two properties make end-to-end tests especially exposed. They cross process boundaries (test, browser, server, database), so every hop has latency that varies, and they run against a user interface that is asynchronous by nature. A suite that treats the page as if each command completed instantly will flake. The defence is the same in every tool: synchronise on state, not on time.

How each tool defends against flakiness

Defence Playwright Cypress Selenium

Waiting for an element before an action

Built in. Every action runs actionability checks (attached, visible, stable, receives events, enabled, editable) and retries until the timeout.

Built in. Commands such as click() and type() wait for the element and for it to be actionable before acting.

Not built in. Use an explicit wait (WebDriverWait) or a page object that waits. Implicit waits exist but should not be mixed with explicit ones.

Waiting for a condition in an assertion

Web-first assertions (await expect(locator)...) retry until the condition holds or the timeout expires.

.should() retries the query chain and the assertion until it passes (retry-ability).

wait.until(condition) polls a condition; there is no retrying assertion.

Retry of a stale lookup

Locators are lazy and re-resolve at every use.

Queries re-run from the start of the chain when the DOM changes.

A found WebElement can go stale; find it again, or wait on a condition that finds it.

Clean state per test

A new browser context per test by default.

The page and cookies are cleared between tests; cy.session() caches a login.

You decide: a new driver per test is the most robust and the slowest.

Retrying a failed test

retries, --retries, per-group and per-project, with a flaky status.

retries for run and open mode, per test or suite, plus experimental strategies.

Not part of WebDriver. Build tools and runners add it: Maven Surefire, JUnit Pioneer, pytest-rerunfailures.

Repeating a test to expose a flake

--repeat-each

No flag; generate the repetitions in the spec or loop in the shell.

JUnit @RepeatedTest, pytest-repeat.

Evidence of a failure

Trace with a timeline, DOM snapshots and network; screenshots and video.

Screenshots per failed attempt, video, and Test Replay in Cypress Cloud.

Screenshots you take yourself; logs and BiDi events; grid recordings.

Source: Playwright: Auto-waiting

Detail on the waiting rules is in the linked tool pages. The point for this page is the gap: Playwright and Cypress wait for you, Selenium makes you wait explicitly, and every tool lets you opt out of the protection by reading the DOM once or by sleeping. The experiment below does exactly that.

A flaky test, reproduced

Each version of the test submits a review for the book with id 3 as the signed-in reader. The flaky version waits 150 ms and then reads the state of the last review row. The fixed version waits until the row containing its own text has the state saved.

Playwright

// tests/review-timing.spec.ts (excerpt)
import { test, expect } from '@playwright/test';

const password = process.env.BOOKSHELF_READER_PASSWORD ?? 'reader-local-only';

test.beforeEach(async ({ page }) => {
  // Sign in through the API; the cookie lands in the test's browser context.
  await page.request.post('/api/session', { data: { username: 'reader', password } });
  await page.goto('/books/3');
});

async function submitReview(page, text: string) {
  await page.getByTestId('review-rating-input').selectOption('4');
  await page.getByTestId('review-text-input').fill(text);
  await page.getByTestId('review-submit').click();
}

// FLAKY: a fixed sleep tuned on a fast machine. The server needs 0-300 ms to save.
test('review is saved (sleep 150 ms)', {
  tag: '@quarantine',
  annotation: { type: 'issue', description: 'Race on the save; replace the sleep with a web-first assertion' },
}, async ({ page }, testInfo) => {
  await submitReview(page, `Flaky sleep review ${testInfo.workerIndex}-${testInfo.repeatEachIndex}-${Date.now()}`);
  await page.waitForTimeout(150);
  const state = await page.getByTestId('review').last().getAttribute('data-state');
  expect(state).toBe('saved');
});

// FIXED: a web-first assertion retries until the row reaches the saved state.
test('review is saved (web-first assertion)', async ({ page }, testInfo) => {
  const text = `Fixed review ${testInfo.workerIndex}-${testInfo.repeatEachIndex}-${Date.now()}`;
  await submitReview(page, text);
  await expect(page.getByTestId('review').filter({ hasText: text })).toHaveAttribute('data-state', 'saved');
});

Source: Playwright: Use web-first assertions

The flaky test fails for two reasons that are easy to miss in review. waitForTimeout is a guess, and getAttribute is a one-shot read that returns whatever the DOM says at that instant. The configuration enables fullyParallel, trace: 'on-first-retry' and retries: 0, so every failure is visible. --repeat-each runs every selected test N times, which turns an occasional flake into a measurable rate:

$ npx playwright test -g sleep --repeat-each=20
  ...
  10 failed
    [chromium] › tests/review-timing.spec.ts:18:5 › review is saved (sleep 150 ms) @quarantine
    ...
  10 passed (3.7s)

$ npx playwright test -g web-first --repeat-each=20
  20 passed (4.7s)

Source: Playwright: Command line

Ten of twenty runs of the sleep version failed with Expected: "saved" and Received: "pending", which is what the 0 to 300 ms delay predicts around a 150 ms sleep. The web-first version passed 20 of 20. The counts differ from run to run (a later run of 5 repetitions happened to pass completely), which is the nature of a flake: a single green run proves nothing, and a repeat count is the way to measure it. --repeat-each is also documented under parallelism in Parallelism, Sharding & Retries; combine it with --workers to put the machine under the load that reveals races.

Cypress

Cypress has no repeat flag. The usual approaches are to generate the tests in a loop, as below, to run the same spec several times from a shell loop or a CI matrix, or to add a plugin such as cypress-repeat. The loop approach needs no dependency and keeps every repetition as a separate named test.

// cypress/e2e/review-timing.cy.ts (excerpt)
// Cypress has no --repeat-each: generate the repetitions in the spec.
const repeats = 10;

function submitReview(text: string) {
  cy.get('[data-testid=review-rating-input]').select('4');
  cy.get('[data-testid=review-text-input]').type(text);
  cy.get('[data-testid=review-submit]').click();
}

describe('review save timing', () => {
  beforeEach(() => {
    cy.env(['READER_PASSWORD']).then(({ READER_PASSWORD }) => {
      cy.request('POST', '/api/session', { username: 'reader', password: READER_PASSWORD });
    });
    cy.visit('/books/3');
  });

  Cypress._.times(repeats, (i) => {
    // FLAKY: cy.wait(ms) is a sleep, and .then() runs once, so nothing retries.
    it(`review is saved (sleep 150 ms) #${i + 1}`, () => {
      submitReview(`Flaky sleep review ${i}-${Date.now()}`);
      cy.wait(150);
      cy.get('[data-testid=review]').last().then(($li) => {
        expect($li.attr('data-state')).to.eq('saved');
      });
    });

    // FIXED: .should() retries the query and the assertion until it passes.
    it(`review is saved (retrying assertion) #${i + 1}`, () => {
      const text = `Fixed review ${i}-${Date.now()}`;
      submitReview(text);
      cy.contains('[data-testid=review]', text).should('have.attr', 'data-state', 'saved');
    });
  });
});

Source: Cypress: Retry-ability

The configuration sets retries to 0 (retries: { runMode: 0, openMode: 0 }) and reads the password from BOOKSHELF_READER_PASSWORD. The trap in the flaky version is subtle. cy.get() does retry, but only until the element exists, and the pending row exists immediately. .then() runs its callback once and never retries, so the assertion inside it sees the first state. .should() re-runs the whole query chain, so it waits for the state to change:

$ npx cypress run --browser electron
  review save timing
    ✓ review is saved (sleep 150 ms) #1 (649ms)
    ✓ review is saved (retrying assertion) #1 (560ms)
    1) review is saved (sleep 150 ms) #2
    ✓ review is saved (retrying assertion) #2 (592ms)
    ...
    2) review is saved (sleep 150 ms) #7
    ✓ review is saved (retrying assertion) #7 (599ms)
    ...
  18 passing (11s)
  2 failing

  1) review save timing
       review is saved (sleep 150 ms) #2:

      AssertionError: expected 'pending' to equal 'saved'

Source: Cypress: cypress run

Two of ten sleep runs failed and all ten retrying-assertion runs passed. Cypress’s own overhead per command is larger than Playwright’s, so the same 150 ms sleep failed less often here; the failure rate depends on every step in between, which is why flakiness is so hard to predict from the code.

Selenium with Java and JUnit 5

Selenium does not wait for you. The flaky version sleeps and reads once; the fixed version uses an explicit wait with a condition that identifies this test’s own row. @RepeatedTest is the built-in way to repeat a test in JUnit 5.

// ReviewTimingTest.java (excerpt)
  private void submitReview(String text) {
    new Select(driver.findElement(By.cssSelector("[data-testid=review-rating-input]"))).selectByValue("4");
    driver.findElement(By.cssSelector("[data-testid=review-text-input]")).sendKeys(text);
    driver.findElement(By.cssSelector("[data-testid=review-submit]")).click();
  }

  // FLAKY: Thread.sleep() guesses how long the save takes; findElement() reads once.
  @Tag("quarantine")
  @RepeatedTest(10)
  void reviewIsSavedSleep(RepetitionInfo info) throws Exception {
    submitReview("Flaky sleep review " + info.getCurrentRepetition() + " " + System.nanoTime());
    Thread.sleep(150);
    var rows = driver.findElements(By.cssSelector("[data-testid=review]"));
    assertEquals("saved", rows.get(rows.size() - 1).getDomAttribute("data-state"));
  }

  // FIXED: an explicit wait polls until the row of this review reaches the saved state.
  @RepeatedTest(10)
  void reviewIsSavedExplicitWait(RepetitionInfo info) {
    String text = "Fixed review " + info.getCurrentRepetition() + " " + System.nanoTime();
    submitReview(text);
    By saved = By.xpath("//li[@data-testid='review'][@data-state='saved'][contains(., '" + text + "')]");
    new WebDriverWait(driver, Duration.ofSeconds(5))
        .until(ExpectedConditions.visibilityOfElementLocated(saved));
  }

Source: Selenium: Explicit wait

The @BeforeEach method starts headless Chrome, signs in with POST /api/session, copies the session cookie into the browser (the browser must be on a page of the same host first, here /signin) and opens /books/3. One run of mvn -Drerun=0 test printed:

[ERROR] Tests run: 20, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 33.69 s <<< FAILURE! -- in demo.ReviewTimingTest
[ERROR] demo.ReviewTimingTest.reviewIsSavedSleep(RepetitionInfo)[5] -- Time elapsed: 2.102 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <saved> but was: <pending>

Source: JUnit: Repeated tests

Three of ten sleep repetitions failed (repetitions 5, 9 and 10) and the ten explicit-wait repetitions all passed.

Selenium with Python and pytest

# tests/test_review_timing.py (excerpt)
def submit_review(driver, text):
    Select(driver.find_element(By.CSS_SELECTOR, "[data-testid=review-rating-input]")).select_by_value("4")
    driver.find_element(By.CSS_SELECTOR, "[data-testid=review-text-input]").send_keys(text)
    driver.find_element(By.CSS_SELECTOR, "[data-testid=review-submit]").click()


# FLAKY: time.sleep() guesses how long the save takes; find_elements() reads once.
@pytest.mark.quarantine
def test_review_is_saved_sleep(driver):
    submit_review(driver, f"Flaky sleep review {time.time_ns()}")
    time.sleep(0.15)
    rows = driver.find_elements(By.CSS_SELECTOR, "[data-testid=review]")
    assert rows[-1].get_dom_attribute("data-state") == "saved"


# FIXED: an explicit wait polls until this review's row reaches the saved state.
def test_review_is_saved_explicit_wait(driver):
    text = f"Fixed review {time.time_ns()}"
    submit_review(driver, text)
    saved = (By.XPATH, f"//li[@data-testid='review'][@data-state='saved'][contains(., '{text}')]")
    WebDriverWait(driver, 5).until(EC.visibility_of_element_located(saved))

Source: Selenium: Explicit wait

pytest-repeat adds --count (version 0.9.4 was used):

$ pytest tests --count=10 -q
...
FAILED tests/test_review_timing.py::test_review_is_saved_sleep[1-10] - Assert...
FAILED tests/test_review_timing.py::test_review_is_saved_sleep[4-10] - Assert...
FAILED tests/test_review_timing.py::test_review_is_saved_sleep[7-10] - Assert...
3 failed, 17 passed in 27.62s

$ pytest tests -k explicit --count=10 -q
..........
10 passed, 10 deselected in 18.11s

Source: pytest-repeat

Shared state: a flake that parallelism creates

The second classic cause needs no sleep at all. This Playwright spec counts the reviews of a book on the server before and after its own save. With one worker it always passes. With several workers, another test adds a review in between.

// tests/shared-state.spec.ts (excerpt)
// FLAKY: counts every review of the book on the server, which other workers also change.
test('shared state: the server has one more review (global count)', async ({ page, request }, testInfo) => {
  const before = (await (await request.get('/api/reviews?bookId=3')).json()).length;
  await addReview(page, `Counting review ${testInfo.repeatEachIndex}-${Date.now()}`);
  const after = (await (await request.get('/api/reviews?bookId=3')).json()).length;
  expect(after).toBe(before + 1);
});

// FIXED: look only at the data this test created.
test('shared state: the server has my review (own data)', async ({ page, request }, testInfo) => {
  const text = `Own review ${testInfo.repeatEachIndex}-${Date.now()}`;
  await addReview(page, text);
  const reviews = await (await request.get('/api/reviews?bookId=3')).json();
  expect(reviews.filter((r: { text: string }) => r.text === text)).toHaveLength(1);
});

Source: Playwright: Isolate test data between parallel workers

$ npx playwright test shared-state --repeat-each=10 --workers=1
  20 passed (9.3s)
$ npx playwright test shared-state --repeat-each=10 --workers=4
Running 20 tests using 4 workers
  6 failed
  14 passed (5.6s)
    Expected: 714
    Received: 716
$ npx playwright test shared-state -g "own data" --repeat-each=10 --workers=4
  10 passed (2.9s)

Source: Playwright: Command line

The failure message is the signature of shared state: the count is off by a small, changing amount. The fix is the general rule: assert on data the test created and can identify, not on totals that anything else can change. Bookshelf’s /api/test/reset endpoint is a trap for the same reason, because it also clears every session; the reasoning is in Playwright: Authentication & Isolation and the Cypress equivalent, cy.session() with a validate function, is in Cypress: Sessions, Authentication & Cross-Origin.

Retries and flaky classification

A retry runs a failed test again. It does not remove flakiness; it separates flaky tests (failed, then passed) from broken tests (failed every time) and keeps one unlucky run from blocking a merge. The risk is that a retry hides the problem, so each tool should report the retry.

Playwright

Playwright classifies a test as passed, flaky (failed then passed on a retry) or failed. The configuration is retries (a number), the --retries flag, test.describe.configure() for a group and testProject.retries for a project; the full behaviour (the worker restart, retryStrategy, testInfo.retry, --last-failed) is in Parallelism, Sharding & Retries. On the racy test:

$ npx playwright test -g sleep --retries=2 --repeat-each=6
  ✓  1 [chromium] › tests/review-timing.spec.ts:18:5 › review is saved (sleep 150 ms) (399ms)
  ...
  ✘  2 [chromium] › tests/review-timing.spec.ts:18:5 › review is saved (sleep 150 ms) (461ms)
  ✘  7 [chromium] › tests/review-timing.spec.ts:18:5 › review is saved (sleep 150 ms) (retry #1) (440ms)
  ✓  8 [chromium] › tests/review-timing.spec.ts:18:5 › review is saved (sleep 150 ms) (retry #2) (321ms)
  1 flaky
  5 passed (2.5s)

Source: Playwright: Retries

One repetition failed twice and passed on the second retry, so the run reports 1 flaky and exits with code 0. Because a green exit hides that, Playwright can fail the run on flakiness: --fail-on-flaky-tests, or failOnFlakyTests: true in the configuration:

$ npx playwright test -g sleep --retries=5 --repeat-each=10
  5 flaky
  5 passed (6.1s)
$ echo $?
0
$ npx playwright test -g sleep --retries=5 --repeat-each=10 --fail-on-flaky-tests
$ echo $?
1

Source: Playwright: Command line

A common policy: retries on CI so one unlucky run does not block developers, and --fail-on-flaky-tests in a nightly job (or a required check on the main branch) so the offenders have to be fixed.

Cypress

Cypress retries a failed test in cypress run and, if configured, in cypress open. retries: 2 means up to three attempts; beforeEach and afterEach run again for each attempt, and failures in before and after do not trigger retries. The value can be set globally, per suite or per test:

import { defineConfig } from 'cypress';

export default defineConfig({
  retries: {
    runMode: 2,   // cypress run
    openMode: 0,  // cypress open
  },
});

// Per test or per suite
describe('review save', { retries: { runMode: 2, openMode: 1 } }, () => {
  it('saves the review', () => {
    // ...
  });
});

Source: Cypress: Test retries

The configuration above is the documentation’s and was not run as shown. The command line override that was run is --config retries=2; a dotted key such as retries.runMode=2 was rejected by Cypress 16.2.0 with "The following configuration option is invalid".

$ npx cypress run --browser electron --config retries=2
  review save timing
    ✓ review is saved (sleep 150 ms) #5 (505ms)
    (Attempt 1 of 3) review is saved (sleep 150 ms) #6
    ✓ review is saved (sleep 150 ms) #6 (520ms)
    ...
  20 passing (13s)

Source: Cypress: Test retries

Three tests were retried and the run passed with exit code 0. The terminal lines for retries are easy to overlook, and the summary does not say "flaky". Cypress keeps a screenshot of each failed attempt, named with the attempt number, for example ... (failed) (attempt 2).png. To fail the run when a flake is detected, use the experimental retry strategies, which give control over the final result:

  • detect-flake-and-pass-on-threshold with maxRetries and passesRequired: pass only when enough attempts pass.

  • detect-flake-but-always-fail with maxRetries and stopIfAnyPassed: any test with a failed attempt ends as failed, and a flaky one is also reported as flaky.

Both require openMode and runMode to be set to true or false, not numbers, and they apply globally, not per test. This configuration (cypress.always-fail.config.ts, passed with --config-file) was run:

import { defineConfig } from 'cypress';

export default defineConfig({
  e2e: {
    baseUrl: process.env.BASE_URL ?? 'http://localhost:3966',
    video: false,
    supportFile: false,
  },
  retries: {
    experimentalStrategy: 'detect-flake-but-always-fail',
    experimentalOptions: { maxRetries: 2, stopIfAnyPassed: true },
    openMode: false,
    runMode: true,
  },
  env: { READER_PASSWORD: process.env.BOOKSHELF_READER_PASSWORD ?? 'reader-local-only' },
});

Source: Cypress: Experimental test retries

$ npx cypress run --browser electron --config-file cypress.always-fail.config.ts
    ✖(Attempt 1 of 3) review is saved (sleep 150 ms) #3
    ✓(Attempt 2 of 3) review is saved (sleep 150 ms) #3
    ✖ review is saved (sleep 150 ms) #3 (685ms)
    ...
  15 passing (15s)
  5 failing
$ echo $?
5

Source: Cypress: Experimental test retries

Every flaky test was reported as failed (the final result of #3 is the cross), the five failures are the five flakes, and the exit code is the number of failed tests. This is Cypress’s equivalent of Playwright’s --fail-on-flaky-tests.

Selenium: the test runner adds retries

WebDriver has no concept of a test, so retry support comes from the runner. Three options were run.

Maven Surefire reruns failing tests with rerunFailingTestsCount (JUnit 4.12+, JUnit 5 and TestNG) and reports flakes in its summary. The POM reads the count from a property:

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.5.4</version>
  <configuration>
    <!-- Re-run a failing test up to N more times; a pass after a failure is reported as a flake. -->
    <rerunFailingTestsCount>${rerun}</rerunFailingTestsCount>
  </configuration>
</plugin>

Source: Maven Surefire: Rerun failing tests

$ mvn -Drerun=2 test
[ERROR] Failures:
[ERROR] demo.ReviewTimingTest.reviewIsSavedSleep(RepetitionInfo)[1]
[ERROR] demo.ReviewTimingTest.reviewIsSavedSleep(RepetitionInfo)[7]
[WARNING] Flakes:
[WARNING] demo.ReviewTimingTest.reviewIsSavedSleep(RepetitionInfo)[3]
[INFO]   Run 2: PASS
[ERROR] Tests run: 20, Failures: 2, Errors: 0, Skipped: 0, Flakes: 1
[INFO] BUILD FAILURE

Source: Maven Surefire: Rerun failing tests

Repetition 3 is listed under Flakes (failed once, passed on rerun) and the build still counts repetitions 1 and 7, which failed on all three runs (the first run and two reruns), as failures. A flake does not fail the Surefire build by default.

JUnit Pioneer provides @RetryingTest, which retries inside the test method without a build-plugin setting:

// Retry inside the test method itself: JUnit Pioneer re-runs it up to 3 times (no Surefire setting needed).
@RetryingTest(maxAttempts = 3, minSuccess = 1)
void reviewIsSavedSleepWithPioneerRetry() throws Exception {
  submitReview("Pioneer sleep review " + System.nanoTime());
  Thread.sleep(150);
  var rows = driver.findElements(By.cssSelector("[data-testid=review]"));
  assertEquals("saved", rows.get(rows.size() - 1).getDomAttribute("data-state"));
}

Source: JUnit Pioneer: Retrying Failing Tests

In the run, Tests run: 2, Failures: 0, Errors: 0, Skipped: 1: Pioneer executed the test twice, reports the failed first attempt as skipped, and the test passed on the second. Pioneer 2.3.0 needed an explicit junit-platform-launcher dependency at the platform version that matches JUnit (1.14.4 for JUnit Jupiter 5.14.4); without it the Jupiter engine failed to discover tests. minSuccess and suspendForMs control how many passes are needed and the pause between attempts.

pytest-rerunfailures (version 16.7, Python 3.10+) re-runs failed tests and prints them with -r R:

$ pytest tests -k sleep --count=10 --reruns 2 -r R -q
RERUN tests/test_review_timing.py::test_review_is_saved_sleep[5-10]
FAILED tests/test_review_timing.py::test_review_is_saved_sleep[5-10]
RERUN tests/test_review_timing.py::test_review_is_saved_sleep[6-10]
PASSED tests/test_review_timing.py::test_review_is_saved_sleep[6-10]
...
1 failed, 9 passed, 10 deselected, 8 rerun in 18.02s

Source: pytest-rerunfailures: Re-run all failures

Repetition 6 was re-run and passed (a flake that the summary reports only as "rerun"), and repetition 5 failed all its attempts. The plugin also offers --reruns-delay, --only-rerun and --rerun-except (filter by exception text, so only assertion failures are retried and an unavailable driver is not), and a per-test marker such as @pytest.mark.flaky(reruns=2).

Choosing a retry policy

Tool How a flake appears How to make it count

Playwright

N flaky in the summary; exit code 0.

--fail-on-flaky-tests or failOnFlakyTests: true.

Cypress

(Attempt 1 of 3) lines; the run passes. Cypress Cloud flags the run.

Experimental detect-flake-but-always-fail; Cypress Cloud flaky alerts.

Selenium and JUnit

Surefire Flakes: section; Pioneer skipped attempts.

Treat a non-empty Flakes section as a CI signal; Surefire itself does not fail on it.

Selenium and pytest

RERUN lines with -r R; the final report counts reruns.

Fail the job on a non-zero rerun count with a small script or a CI check.

Three rules keep retries honest: retry only in CI (a developer should see the first failure), keep the count low (1 or 2), and record every flake somewhere visible.

Cypress Cloud flaky test management

Cypress Cloud turns the retry data of recorded runs into a flake dashboard: a run with flaky tests is flagged in Latest Runs, each test has a flake severity and a history (flake rate and failure rate over time), and alerts reach GitHub pull request comments, GitLab and Bitbucket commit statuses (cypress/flake), Slack and Microsoft Teams. Flake detection, analytics and alerting need a Team plan or higher. It depends on test retries being on: Cloud sees a flake because attempt 1 failed and a later attempt passed. Test Replay (available on all Cloud plans, subject to usage limits) lets you open a flaky run and compare a passing attempt with a failing one, with the DOM, network and console of each. The details of recording, plans and the other Cloud features are in Cypress Cloud: flaky test management.

Source: Cypress Cloud: Flaky Test Management

None of this was run: it needs a Cypress Cloud account and a recorded project. Playwright has no hosted equivalent in the open-source runner; its flake data comes from the reporters (the json and blob reporters list each test’s attempts), and Azure App Testing and third-party dashboards build trends on top of them.

Quarantine

A quarantine moves a known-flaky test out of the blocking run while keeping it visible and running somewhere, so the suite stays trustworthy and the flake does not get forgotten. A test that is simply skipped is lost; a quarantined test has an owner, a ticket and a deadline.

The mechanism is a tag plus a filter. In Playwright the test above carries tag: '@quarantine' and an annotation with a description, and the blocking job excludes the tag while a separate job runs only the quarantined tests:

$ npx playwright test --repeat-each=5 --grep-invert @quarantine
  5 passed (1.3s)
$ npx playwright test --grep @quarantine --repeat-each=5
  2 passed (1.7s)
$ npx playwright test --list --grep @quarantine
  [chromium] › review-timing.spec.ts:18:5 › review is saved (sleep 150 ms)
Total: 1 test in 1 file

Source: Playwright: Tag tests

The first command ran only the five repetitions of the stable test (the blocking run). The second ran the quarantined test five times and, by chance, passed 2 and failed 3 (the failure lines are omitted). Other Playwright options are test.fixme() and test.skip() with a reason, which show up as skipped, and test.fail(), which expects the test to fail and starts failing the build when the underlying problem is fixed. The same idea in the other stacks:

# JUnit 5: @Tag("quarantine") on the test, excluded from the blocking build
$ mvn -B -Drerun=0 -DexcludedGroups=quarantine test
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS

# pytest: @pytest.mark.quarantine, registered in pytest.ini
$ pytest tests -m "not quarantine" -q
1 passed, 1 deselected in 2.11s
$ pytest tests -m quarantine --count=3 -q
1 failed, 2 passed, 3 deselected in 4.08s

Source: pytest: Working with custom markers

For Cypress, put quarantined specs in their own folder and run them with --spec, or use it.skip() with a comment and a ticket reference; the @cypress/grep plugin adds tag filtering. Neither Cypress approach was run here.

A workable policy: quarantine within a day of detecting a flake, assign an owner, run the quarantine job on every merge without blocking, fix or delete within a deadline, and report the quarantine size as a health metric.

Diagnosing a flake

The goal is to see what the browser showed at the moment of failure, not to re-run until it goes away.

Playwright traces

trace: 'on-first-retry' records a trace of the first retry. If the retry passes (a flake), the trace shows the passing attempt, which is useful for comparison but does not show the failure. To capture the failing attempt use retain-on-failure (or retain-on-first-failure, on-all-retries); the cost is the disk space of traces for failures. The trace can be opened in the Trace Viewer (npx playwright show-trace, not run here because it opens a window). The command line also offers npx playwright trace, which inspects a trace from the terminal. It was run on a failing repetition recorded with --trace=retain-on-failure:

$ npx playwright test -g sleep --repeat-each=10 --trace=retain-on-failure
$ npx playwright trace open test-results/review-timing-review-is-saved-sleep-150-ms--chromium-repeat6/trace.zip
$ npx playwright trace actions
  ...
   13. 0:00.675  Click                                                       44ms
                 getByTestId('review-submit')
   14. 0:00.720  Wait for timeout                                           211ms
   15. 0:00.932  Get attribute "data-state"                                  11ms
                 getByTestId('review').last()
   16. 0:00.945  Expect "toBe"                                                3ms  ✗
$ npx playwright trace errors
  Expected: "saved"
  Received: "pending"
$ npx playwright trace close

Source: Playwright: Trace viewer

The action list tells the story: a click, a 211 ms Wait for timeout, a one-shot attribute read and a failed toBe. Nothing waits for the application. The network view of the same trace lists the POST /api/reviews request as still in flight (state ERR, because the browser closed with the request unfinished). More on traces, screenshots and video is in Screenshots, Videos & Traces and Playwright: Debugging, Tracing & Codegen. AI-assisted repair of failing Playwright tests, which includes a "healer" agent, is covered in Playwright Test Agents.

Cypress screenshots and Test Replay

Locally, Cypress saves a screenshot for each failed attempt (the run above wrote them for the retried tests) and can record a video. Time-travel in cypress open shows the Command Log of each attempt. In CI the best evidence is Test Replay in Cypress Cloud, which replays the failed attempt with the DOM, network requests, console and rendering as they were in CI; it was not run here (see the Cloud section). The screenshot and video options and the Studio and reporter tools are in Cypress: Debugging, Studio, Plugins & Reporters.

Selenium evidence

Selenium has no built-in trace. Collect evidence yourself: a screenshot and the page source in a JUnit extension or a pytest hook on failure, browser console logs, WebDriver BiDi events (see WebDriver BiDi) and, on Selenium Grid, the session recording of the node image. Selenium’s own guidance is to treat flakiness as a synchronisation bug first: replace sleeps with explicit waits.

A diagnosis checklist

  • Reproduce: run the single test 20 to 50 times (--repeat-each, a loop, @RepeatedTest, --count). Add load with more workers or a CPU-limited container.

  • Read the failing attempt, not the passing retry: the first error message is usually the cause.

  • Look for the guess: a sleep, a one-shot read, a global count, a fixed date.

  • Compare passing and failing runs with traces or Test Replay.

  • Check the environment: same browser version, same image, same viewport and number of workers as CI.

  • Rule out a product bug: if the page really shows an inconsistent state, fix the application.

Fixing the cause

The fix for the example is the same idea in every tool: replace a guess about time with a wait for the condition, and look only at data the test owns. The general toolbox:

Problem Fix

Sleep or fixed delay

Playwright web-first assertion or expect.poll(); Cypress .should(); Selenium WebDriverWait with an expected condition. Delete waitForTimeout, cy.wait(ms) (waiting on an aliased request is fine), Thread.sleep and time.sleep.

One-shot read of the DOM

Use the retrying form: expect(locator) instead of await locator.getAttribute() followed by expect(value); .should() instead of .then(); a wait condition instead of findElements.

Shared or global data

Unique data per test, API seeding, a user per worker, assertions about your own rows.

External services

Mock them (page.route, cy.intercept) or run a stub; see Playwright: Network Mocking, Clock & API Testing and Cypress: Network, Stubs, Spies & Clocks.

Animations

Wait for actionability, disable animations in the test build, or mask them in visual tests.

Overloaded runner

Fewer workers, more CPU, the isolated retry strategy in Playwright 1.62 or later, sharding across machines.

After a fix, repeat the test as many times as it failed before. A fix that passes 20 of 20 under the same load that failed 10 of 20 is evidence; one green run is not. Remove the quarantine tag in the same change.

Pitfalls

  • Retries as a cure. A retry turns a red build green and the flake stays. Count flakes and fail on them periodically.

  • Raising timeouts. A longer timeout makes the suite slower and the flake rarer, not gone. Find the missing condition.

  • Retrying the whole job. Re-running a pipeline until it passes hides the same problem and wastes minutes.

  • Mixing implicit and explicit waits in Selenium. The waits add up unpredictably; use explicit waits only.

  • Hiding flakes with --workers=1. It removes shared-state flakes by making the suite slow, and the flake returns the day someone raises the worker count.

  • Quarantine without an owner. A quarantine list that only grows is a graveyard of coverage.

  • Treating headless and headed as equal. Timing, focus and animations differ; run the same mode locally and in CI.

What changed recently

  • Playwright 1.45 and 1.52: --fail-on-flaky-tests and the failOnFlakyTests option. 1.62 added retryStrategy (immediate or isolated) and 1.64 added --shuffle. The npx playwright trace command line inspects traces without the viewer window.

  • Cypress 16.0 (1 September 2026): Cypress.env() was removed in favour of cy.env() and Cypress.expose(). In 16.2.0 the Electron browser prints a deprecation warning at the start of a run that advises switching to Chrome or another installed browser. Experimental test retries (detect-flake-and-pass-on-threshold and detect-flake-but-always-fail) remain experimental at the check date.

  • Selenium 4.51: Java ExpectedCondition no longer extends Guava’s Function; the ExpectedConditions factory used in the example above ran unchanged.

  • JUnit Pioneer 2.3.0 needs the JUnit Platform launcher to match the Jupiter version; pytest-rerunfailures 16.7 and pytest-repeat 0.9.4 were the current releases at the check date.

References