Rayrun
โ† Back to Discord Forum

Images and toHaveScreenshot best practice

Hi!

I'm just getting started with Playwright, and I figured I'd try to set up an e2e test stack mostly based on visual comparisons. I'm not sure if this is a good idea, but I have very good experience using a snapshot based e2e testing approach while working with pure backend.

One issue I'm running into is that the app has quite a bit of images loaded from external sources. It seems quite a bit of the time, the screenshot is taken before the images finish loading. Is there a recommended way to deal with this?

This thread is trying to answer question "What is the recommended way to deal with screenshots being taken before images finish loading in Playwright?"

17 replies
@.petob: Hey, if you want to wait for the images to finish loading you can increase the timeout for the toHaveScreenshot function https://playwright.dev/docs/api/class-pageassertions#page-assertions-to-have-screenshot-2-option-timeout If on the other hand you do not want to wait for all the images to finish loading you might be interested in masking the images with https://playwright.dev/docs/api/class-pageassertions#page-assertions-to-have-screenshot-2-option-mask using a locator for your images that should not be included in the comparison because they have not been fully loaded. However, this would require that the rest of your page looks the same if the pictures are not finished loading

Hey, if you want to wait for the images to finish loading you can increase the timeout for the toHaveScreenshot function https://playwright.dev/docs/api/class-pageassertions#page-assertions-to-have-screenshot-2-option-timeout

If on the other hand you do not want to wait for all the images to finish loading you might be interested in masking the images with https://playwright.dev/docs/api/class-pageassertions#page-assertions-to-have-screenshot-2-option-mask using a locator for your images that should not be included in the comparison because they have not been fully loaded. However, this would require that the rest of your page looks the same if the pictures are not finished loading

Timeout seems nice, but a problem I'm running into now is that the first screenshots sometimes are generated wrong without the images, and due to flakiness it seems to be basically impossible to generate everyone right at the same time...

Masking seems interesting, thanks!

I was also wondering if I could use something

await page.waitForLoadState("networkidle");

to ensure external images are finished loading but it doesn't seem to work

Hi @.petob Browsers know how to render images. You don't need to test that ๐Ÿ˜‰ what matters is the layout of your page/app. Therefore you should add a route to replace any network call to images to return a placeholder image, transparent image, gradient,... whatever makes most sense for you.

This will speed up your tests, remove some flakiness due to external requests and additional risks of visual noise.

@p01_ Good point! Do you have any pointers on how to do this?

Masks are not great because they overlay a color on top the elements they mask, so any change of dimension of that element will make your tests fail even if the masked elements don't effet the layout of the page

https://playwright.dev/docs/api/class-page#page-route

You can do a route per domain hosting the images, or instead of specifying the url as a string, use a method and do some matching yourself eg against an array of domains and path hosting your images

We use a function matching the url against the domain and path of our image hosts in our E2E and Component tests. It does wonders.

~6000 screenshots and not a single flake due to loading images

Nice, thanks for the pointer! Is this done in playwrights internal tests?

I don't think so. Playwright's own visual tests are very specific and don't deal with external resources to test the API themselves.

But I don't work on Playwright. Just at Microsoft, on a monorepo.

Aha, thanks anyways!

Hope you manage to solve the problem.

If needed, I can share a snippet tomorrow.

@p01_: That would be very nice!

That would be very nice!

import { createHash } from "crypto";

const PLACEHOLDER_IMAGE_HOSTNAMES = [
  "picsum.photos/foo/",
  "media.tenor.com/",
];

// Route placeholder image requests for fast and deterministic response
await page.route((url) => {
  const hostNameAndPathName = url.hostname + url.pathname;
  return PLACEHOLDER_IMAGE_HOSTNAMES.some((v) => hostNameAndPathName.startsWith(v));
}, (route, request) => {
  // Create hash (32chars long) of the URL to grab some hex values for the SVG
  const url = request.url();
  const hash = createHash("md5").update(url).digest("hex");
  const body = `<svg width="100" height="100" xmlns="http://www.w3.org/2000/svg">
    <rect width="100%" height="100%" fill="#${hash.substring(0, 6)}" />
    <rect width="80%" height="100%" fill="#${hash.substring(6, 12)}" />
    <rect width="60%" height="100%" fill="#${hash.substring(12, 18)}" />
    <rect width="40%" height="100%" fill="#${hash.substring(18, 24)}" />
    <rect width="20%" height="100%" fill="#${hash.substring(24, 30)}" />
    text y="50" font-size="50">${hash.substring(30, 32)}</text>
  </svg>`;
  return route.fulfill({ body, contentType: "image/svg+xml" });
});

Something like that ๐Ÿ˜‰ It replaces the requests to some host+path by a deterministic SVG image based on a hash of the original url

Great, thanks a ton! While I have you here, do you have any other recommendations wrt. best practices? The approach I'm thinking now in the start is just to use playwright to navigate through the app and do the "main flows", and finally take a screenshot at the end of the flow. I figure this way I will pretty easily notice if something important breaks, and I will be forced to acknowledge changes to UI in each PR

Yep, that's a nice way to go. For E2E tests, prefer long tests with multiple soft assertions and a few visual tests to make the most of the sessions in your page/app.

Mock any and all external resources you can to avoid flakes and because these either don't make sense to test (eg images in a browser) or should already be tested (eg backend and API calls)

Answer

Sure, you want to take screenshots in Playwright, but you're running into an issue where the images haven't finished loading. No worries, I've got you covered!

Playwright's waitForLoadState method is your best friend here. It lets you wait until the page reaches a certain load state, like "networkidle" or "domcontentloaded", before you take that screenshot.

Here's a quick example:

import { test, expect } from '@playwright/test';

test('example test', async ({ page }) => {
  await page.goto('https://ray.run');
  
  // Chill until the network is idle
  await page.waitForLoadState('networkidle');
  
  // Now, take that screenshot!
  await expect(page).toHaveScreenshot();
});

In this snippet, we first head over to a webpage with page.goto. We then tell Playwright to hang tight with page.waitForLoadState until there's no more than two network connections for at least 500 milliseconds. This makes sure all network requests are done and dusted.

Once we've waited for the load state we want, we can take our screenshot with expect(page).toHaveScreenshot(). This way, we know we're capturing the webpage in all its fully-loaded glory.

So, by using waitForLoadState in your Playwright tests, you can make sure your screenshots are picture-perfect and don't miss any images. Happy testing!

Related Discord Threads

TwitterGitHubLinkedIn
AboutQuestionsDiscord ForumBrowser ExtensionTagsQA Jobs

Rayrun is a community for QA engineers. I am constantly looking for new ways to add value to people learning Playwright and other browser automation frameworks. If you have feedback, email luc@ray.run.