← Back to Discord Forum

Fail Pipeline only when Test Results reached a maximum Number of Failed and Flaky

I don't know if this is possible in the config? or this is set in the yaml? i don't have an idea how to do it

So basically, i want the pipeline to pass if there's only 2 failed, and if i reached for example 5 failed and 5 flaky tests that will be the time that the job will fail.


This thread is trying to answer question "How can I configure the pipeline to only fail when a certain number of tests have failed or are flaky?"

1 reply

You will need to implement custom reporter and based on your conditions (calculate passed / failed / flaky, etc) you will need to override exit status in onEnd function. Basically, like you said, if only 2 tests failed then you will need to set result status to 'passed'


Configuring Test Retries in Playwright Test

To manage failing or flaky tests, you can use the test retries feature in Playwright Test. By default, failing tests are not retried. But you can change this by specifying the number of retry attempts.

Enabling Test Retries via Command Line

Use the --retries flag followed by the number of retry attempts. Here's how:

npx playwright test --retries=3

Configuring Test Retries in Configuration File

You can also set the number of retry attempts in the playwright.config.ts file.

import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: 3,

Understanding Test Categories

Playwright Test categorizes tests into "passed", "flaky", and "failed". You can detect whether a test is being retried at runtime using testInfo.retry.

Setting Retries for Specific Tests

To specify retries for a specific group of tests or a single file, use test.describe.configure().

import { test } from '@playwright/test';

test.describe(() => {
  test.describe.configure({ retries: 2 });

  test('test 1', async ({ page }) => {
    // ...

  test('test 2', async ({ page }) => {
    // ...

Running Dependent Tests in Order

Use test.describe.serial() to ensure that dependent tests always run together and in order.

import { test } from '@playwright/test';

test.describe.configure({ mode: 'serial' });

test.beforeAll(async () => {
   /* ... */

test('first good', async ({ page }) => {
   /* ... */

test('second flaky', async ({ page }) => {
   /* ... */

test('third good', async ({ page }) => {
   /* ... */

By enabling test retries and configuring them appropriately, you can control how many times failing or flaky tests should be retried. For more on handling flaky tests, check out this post.

Related Discord Threads

AboutQuestionsDiscord ForumBrowser ExtensionTagsQA Jobs

Rayrun is a community for QA engineers. I am constantly looking for new ways to add value to people learning Playwright and other browser automation frameworks. If you have feedback, email luc@ray.run.