How do I use soft assertions and custom error handling to prevent retries on soft assertion failures in flaky tests with Playwright Test?

Answer

Sure, you can prevent retries on soft assertion failures in flaky tests with @playwright/test. Soft assertions are checks that don't stop the test when they fail. They let the test continue and check more things.

To prevent retries, you can use soft assertions and custom error handling. When a soft assertion fails, it won't trigger a retry by default. Instead, you can manually handle the failure and decide whether to continue running further checks or stop the test execution.

Here's how you can do it:

import { expect } from '@playwright/test';

test('my flaky test', async ({ page }) => {
  // Make some soft assertions
  await expect.soft(page.getByTestId('status')).toHaveText('Success');
  await expect.soft(page.getByTestId('eta')).toHaveText('1 day');

  // Check for any soft assertion failures
  if (expect(test.info().errors).toHaveLength(0)) {
    // Continue with additional checks or actions
    await page.getByRole('link', { name: 'next page' }).click();
    await expect.soft(page.getByRole('heading', { name: 'Make another order' })).toBeVisible();
    
    // ... more code ...
  } else {
    // Handle the soft assertion failures
    // ... code for handling failures ...
  }
});

By manually checking for soft assertion failures and deciding how to handle them, you have control over whether retries should occur in flaky tests. If there were no soft assertion failures, the test will continue running without triggering retries. On the other hand, if there were soft assertion failures, you can implement custom error handling logic to handle those specific cases.

For more details, check out Mastering the Art of Detecting and Handling Flaky Tests in Playwright.