← Back to Discord Forum

Is there any way to influence the timing of retries?

I don't know if this is the reason, but it seems like when my tests fail, they fail repeatedly in a short period. I then run them again and they pass first time. My theory is that some tests just don't do especially well when run in the "thick" of all the other tests. What I would really like is only to retry AFTER all other tests have completed. i.e. I want to control the test scheduler. Is this possible?

This thread is trying to answer question "Is there any way to influence the timing of retries in test scheduler?"

9 replies

It sounds like you might be running into a problem with tests interfering with each other, or having dependencies that cause them to fail when run in a certain order.

If you diable parrallism do they pass? Run npx playwright test --workers=1

Is it always the same tests??

yes, it's always a handful of "high difficulty" tests - running a checkout flow that depends on Stripe's response time

If you disable parrallism did they pass?

I'm trying that, but it's not a great solution either because these tests take 30 seconds sometimes, and serializing that is bad. (No argument that the real problem is the system under test)

Yes but it’ll tell you whether there are interdependencies making them fail or whether your timeout needs increasing for certain tests

Someone asked a similar question in that they wanted to retry after all tests were run. There was a resolution for this, sadly do not remember. You could tag these test as flaky and then run them in an adjusted pipeline from your fast running tests not ideal but would give you a bit more control over your tests.

You can try searching the <#1054804523652231198> channel for "retries": I got 22 results, some of which might be what NZCarnage is referring to.

Yeah, I did find relevant discussions. The problem is they all basically run a new Playwright run afterwards. And that's fine, but it's non-optimal because all the reporting would be screwed up and split into multiple sets of reports.

Probably a case for digging into the Playwright code itself and seeing what can be done


Configuring Retry Attempts in Playwright Test

Yes, you can influence the timing of retries in Playwright Test. By default, failing tests aren't retried. But you can specify the number of retry attempts either through the command line or in the configuration file.

Command Line Configuration

Use the --retries flag followed by the desired number of retries. For example, npx playwright test --retries=3 gives failing tests 3 retry attempts.

Configuration File

You can also configure retries in the playwright.config.ts file. Use defineConfig to set the retries property. Here's an example:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  // Give failing tests 3 retry attempts
  retries: 3,

Test Status

Playwright Test categorizes tests into "passed", "flaky", and "failed". "Passed" are tests that passed on the first run. "Flaky" are tests that failed initially but passed when retried. "Failed" are tests that failed on all runs including retries.

Retries at Runtime

You can detect whether a test is being retried or not by checking testInfo.retry. This property is accessible within any test, hook, or fixture.

Serial Mode

Playwright Test provides a serial mode (test.describe.serial()) for grouping dependent tests together. If one test fails in serial mode, all subsequent tests in the group are skipped. All tests within a serial group are retried together.

Isolation of Tests

Playwright Test creates an isolated Page object for each test. If you want to reuse a single Page object between multiple tests, you can create your own Page object in test.beforeAll() and close it in test.afterAll().

For more information, check out this blog post: Exploring the Various Retry APIs of Playwright for Robust Testing.

AboutQuestionsDiscord ForumBrowser ExtensionTagsQA Jobs

Rayrun is a community for QA engineers. I am constantly looking for new ways to add value to people learning Playwright and other browser automation frameworks. If you have feedback, email luc@ray.run.