Rayrun

What are some strategies to handle flaky visual comparison tests in Playwright due to minor changes in image dimensions?

Answer

Handling Flaky Visual Comparison Tests in Playwright

Flaky visual comparison tests can be a pain, especially when they're due to a 1-pixel change in image dimensions. But don't worry, there are several ways to tackle this.

Adjust the Threshold Value

The threshold value represents the acceptable perceived color difference between pixels in compared images. By default, Playwright uses a threshold value of 0.2. You can customize this threshold by passing the threshold option to the toMatchSnapshot() or toHaveScreenshot() method.

import { test } from '@playwright/test';

test('should match screenshot', async ({ page }) => {
  await page.goto('https://ray.run');
  expect(await page.screenshot()).toMatchSnapshot('screenshot.png', { threshold: 0.3 });
});

Modify the maxDiffPixels Option

This option specifies an acceptable amount of pixels that could be different between compared images. By default, it is unset, but you can set a specific value for maxDiffPixels.

import { test } from '@playwright/test';

test('should match screenshot', async ({ page }) => {
  await page.goto('https://ray.run');
  expect(await page.screenshot()).toMatchSnapshot('screenshot.png', { maxDiffPixels: 10 });
});

Adjust the maxDiffPixelRatio Option

This option represents an acceptable ratio of differing pixels relative to the total number of pixels in an image. This allows you to define how much variation in pixel differences should be tolerated.

import { test } from '@playwright/test';

test('should match screenshot', async ({ page }) => {
  await page.goto('https://ray.run');
  expect(await page.screenshot()).toMatchSnapshot('screenshot.png', { maxDiffPixelRatio: 0.05 });
});

Remember, these options should be used with caution and tailored according to your specific requirements. For more tips on handling flaky tests, check out this blog post.

References

Thank you!
Was this helpful?
Still have questions?

If you still have questions, please ask a question and I will try to answer it.

Related Discord Threads

Related Questions

TwitterGitHubLinkedIn
AboutQuestionsDiscord ForumBrowser ExtensionTagsQA Jobs

Rayrun is a community for QA engineers. I am constantly looking for new ways to add value to people learning Playwright and other browser automation frameworks. If you have feedback, email luc@ray.run.