Difference in screenshots taken in docker and gitlab CI with the same image

Hey. Are you able to take screenshots when you are running test in docker?

I am using toHaveScreenshot({}), it is comparing if screenshot is already present but if screenshot is not present it do not take the screenshot

Can you help?

well we were able to update screenshots using --update-snapshots in docker , and we have toHaveScreenshot() in our step that compares screens:

await expect(page.getByTestId(locator)).toHaveScreenshot(`${$testInfo.title}.png`, defaultThresholds);

we haven't ran any scenario yet in docker that hadn't had screens.

Hi, well, it's totally ok because it's 2 different systems and they can have slight changes in browsers. I simply run PW docker image locally and generate screenshots there. I also skipped darwin/win32 screenshots to be able to run these tests locally without docker (and don't commit the same files).

Also make sure PW image and a runner in gitlab CI has the same OS version.

Steps:

1. Run docker run --rm --network host -v $(pwd):/work/ -w /work/ --env-file=$(pwd)/test/config/.env.dev -it mcr.microsoft.com/playwright:v1.39.0-jammy /bin/bash
2. Install dependecies inside docker container
3. Run tests with --update-snapshots flag. I'm using this command "pw:update-snapshots": "yarn pw:run -g snapshot --update-snapshots" and added snapshot postfix to all such tests to easier collect them using --grep flag.

--env-file is optional, of course

Not sure...if you found the solution to this, but I am also currently facing this issue. From what I read online...there is a difference in how graphics are rendered on M1/M2 chips vs Intel chips ....so that causes pixel differences between local and CI. Please lmk if you do find the solution to this.

It is very hard to have the same screenshots if you take them on two different environments. That's one of the reason why I created Argos. All screenshots are taken on CI, then uploaded to Argos (not commited in repo). And then Argos compares it with the good baseline (by using git). It presents several advantages:

• Screenshots are stable (taken on CI)
• Screenshots are not in the repo
• You don't have to run a command locally to update screenshots

Checkout https://argos-ci.com and https://argos-ci.com/playwright if you are interested.

so in the end we've settled on running tests in docker using mcr.microsoft.com/playwright:v1.39.0-jammy locally, and in github as well mcr.microsoft.com/playwright:v1.39.0-jammy.

but sadly there was still discrepancy between images run locally and in github, so I had to make my thresholds less strict, setting them to threshold: 0.01, maxDiffPixelRatio: 0.01 , in order for tests to pass locally in docker and in github. We'll see if this threshold/ratio will bite us in the ass in the future

We are also running in to this issue and setting the max diff pixel ratio did bite us as playwright missed out on a button change in a page.

The docker image, playwright version, chromium version all matches both CI and the Mac environment

Is there any further update on how to resolve this?

Funny thing is we did not face this issue in puppeteer from which we migrated to

Might assert screenshots are a really nice feature with playwright, but screenshots from what i've seen will always have issues. I've seen screen shots that look exactly the same visually and would get "images differ by 'x' pixels" and for me there was no visual difference... Could even generate new golden images and immediately return my tests, out of my 2000+ tests i would see about 5-10% of my tests failing, usually random/different tests, all run in docker... Never have seen a way around it...

Thats unfortunate but I still cannot wrap my head around on why it was almost(occasionally flaky) perfect with puppeteer and docker setup But same doesn’t work with playwright and docker

And to be more specific threshold was set to 0 in puppeteer setup