Answer

Handling Flaky Visual Comparison Tests in Playwright

Flaky visual comparison tests can be a pain, especially when they're due to a 1-pixel change in image dimensions. But don't worry, there are several ways to tackle this.

Adjust the Threshold Value

The threshold value represents the acceptable perceived color difference between pixels in compared images. By default, Playwright uses a threshold value of 0.2. You can customize this threshold by passing the threshold option to the toMatchSnapshot() or toHaveScreenshot() method.

import { test } from '@playwright/test';

test('should match screenshot', async ({ page }) => {
  await page.goto('https://ray.run');
  expect(await page.screenshot()).toMatchSnapshot('screenshot.png', { threshold: 0.3 });
});

Modify the maxDiffPixels Option

This option specifies an acceptable amount of pixels that could be different between compared images. By default, it is unset, but you can set a specific value for maxDiffPixels.

import { test } from '@playwright/test';

test('should match screenshot', async ({ page }) => {
  await page.goto('https://ray.run');
  expect(await page.screenshot()).toMatchSnapshot('screenshot.png', { maxDiffPixels: 10 });
});

Adjust the maxDiffPixelRatio Option

This option represents an acceptable ratio of differing pixels relative to the total number of pixels in an image. This allows you to define how much variation in pixel differences should be tolerated.

import { test } from '@playwright/test';

test('should match screenshot', async ({ page }) => {
  await page.goto('https://ray.run');
  expect(await page.screenshot()).toMatchSnapshot('screenshot.png', { maxDiffPixelRatio: 0.05 });
});

Remember, these options should be used with caution and tailored according to your specific requirements. For more tips on handling flaky tests, check out this blog post.