← Back to Discord Forum

Is it possible to prevent retries on soft assertion failures?

I have flaky tests (yes, I know, I feel shame). We are working on a11y improvements, and I would like to AVOID test retries on a11y failures, which are currently soft expectations. Is this possible?

This thread is trying to answer question "Is it possible to prevent retries on soft assertion failures?"

4 replies

you can prevent retries but for entire test not for only part of the tests test.describe.configure({ retries: 0 });

Problem is I do want a retry if the test fails for a non-soft assertion, such as normal flake.


test.describe('Dont retry these', () => {
  test.describe.configure({retries: 0});
  test('test 1', async ({page}) => {});
  test('test 2', async ({page}) => {});

I get that. It just doesn't help. I want to retry the tests depending on the type of error that occurs. I can hack it up with some checks at the beginning of each test, but I don't like it. I'd rather there was some way inside a test to say either "retry this test" or "the error that happened here is not related to flake"


Sure, you can prevent retries on soft assertion failures in flaky tests with @playwright/test. Soft assertions are checks that don't stop the test when they fail. They let the test continue and check more things.

To prevent retries, you can use soft assertions and custom error handling. When a soft assertion fails, it won't trigger a retry by default. Instead, you can manually handle the failure and decide whether to continue running further checks or stop the test execution.

Here's how you can do it:

import { expect } from '@playwright/test';

test('my flaky test', async ({ page }) => {
  // Make some soft assertions
  await expect.soft(page.getByTestId('status')).toHaveText('Success');
  await expect.soft(page.getByTestId('eta')).toHaveText('1 day');

  // Check for any soft assertion failures
  if (expect(test.info().errors).toHaveLength(0)) {
    // Continue with additional checks or actions
    await page.getByRole('link', { name: 'next page' }).click();
    await expect.soft(page.getByRole('heading', { name: 'Make another order' })).toBeVisible();
    // ... more code ...
  } else {
    // Handle the soft assertion failures
    // ... code for handling failures ...

By manually checking for soft assertion failures and deciding how to handle them, you have control over whether retries should occur in flaky tests. If there were no soft assertion failures, the test will continue running without triggering retries. On the other hand, if there were soft assertion failures, you can implement custom error handling logic to handle those specific cases.

For more details, check out Mastering the Art of Detecting and Handling Flaky Tests in Playwright.

Related Discord Threads

AboutQuestionsDiscord ForumBrowser ExtensionTagsQA Jobs

Rayrun is a community for QA engineers. I am constantly looking for new ways to add value to people learning Playwright and other browser automation frameworks. If you have feedback, email luc@ray.run.