Rayrun
← Back to Discord Forum

Screenshot assertions failing on the CICD i.e. different OS

_vibhanshu_posted in #help-playwright
Open in Discord

I have developed the testcase for screenshot assertion on Windows machine and I am using ubuntu machine for CICD on my local test case is passing but in CICD same snapshot assertion is failing and vice versa with below error:

`Error: Screenshot comparison failed:

Expected an image 250px by 673px, received 250px by 671px. 5766 pixels (ratio 0.04 of all image pixels) are different.`

Has anyone encountered this issue or please suggest a workaround to overcome this issue

here is my code snip: expect(await this.mainForm.screenshot()).toMatchSnapshot('form.png', { maxDiffPixels: 3 });

This thread is trying to answer question "How can a screenshot assertion testcase developed on a Windows machine pass on a Ubuntu machine used for CICD without using Docker?"

11 replies

Hi, for this cas I use conditional check of testInfo.config.projects[i].use.headless. If it evaluates to true i expect page to match screenshot for headless mode, otherwise - for headed. It is kind of bad practice, but at the moment i could not figure out how to make it better This way you will have to have 2 separate screenshots for both modes

Hi, the screenshots are platform-specific, and the file name is composed of (amongst others) also the platform. So when developing tests on Windows, but running on Linux in CI, you have to generate the reference snapshots using Docker and commit those. The following page explains all that in more detail: https://playwright.dev/docs/test-snapshots

@ondra123. I have tried with headless mode that was also falling

@.aleksandaraleksandrov do you have any workaround for this kind of scenario?

daniel_boone1990
daniel_boone1990

Run it in docker, then OS will be the same amongst platforms

@refactoreric and @daniel_boone1990 we have a team so docker needs to be setup to all the members also need to train them how to run it and when to use which is time taking so I am looking for workaround

The error you have posted means your application is not working properly, maybe not always. The content on different OS are rendered differently - yes but in your case the image sizes are just different and docker or not, you would get the same error. There is no workaround for this.

Well maybe there is, add condition checking or some wait before making assertion to see if you at least will get rid of this different size problem.

Other than that, for different rendering, the docker is a workaround.

.aleksandaraleksandrov
.aleksandaraleksandrov

If there is no workaround, just set the maxDiffPixels -> to a bigger number, or just do it with a DiffRatio of 0.05. @skorp32 is right, you should try waiting, might be an animation or bad CSS.

I agree with @refactoreric only way i've ever got screen shots to work as best as possible, even then still get a number of missmatch screenshots. All images are from the same common docker/os, after that any variables, screen size expect differences.

Related Discord Threads

TwitterGitHubLinkedIn
AboutQuestionsDiscord ForumBrowser ExtensionTagsQA Jobs

Rayrun is a community for QA engineers. I am constantly looking for new ways to add value to people learning Playwright and other browser automation frameworks. If you have feedback, email luc@ray.run.