Difference in screenshots taken in docker and gitlab CI with the same image
Hello everyone!
anyone dabbled with visual testing and gitlab CI?
we're currently trying to move our playwright visual tests to gitlab CI, but screenshots that were taken locally in docker on our MacBook M2 are slightly different from the ones that are being generated at gitlab CI.
Locally in docker we're using FROM mcr.microsoft.com/playwright:v1.39.0-jammy and in gitlab CI we're also specifying image: mcr.microsoft.com/playwright:v1.39.0-jammy, yet screenshots are slightly different (not for human eye). We're trying to figure why they are different and how to solve that.
perhaps anyone ran into such issue? or found a good article/guide online?
Many thanks in advance!
This thread is trying to answer question "Why are screenshots taken in Docker and GitLab CI with the same image slightly different, and how can this be resolved?"
well we were able to update screenshots using --update-snapshots in docker , and we have toHaveScreenshot() in our step that compares screens:
await expect(page.getByTestId(locator)).toHaveScreenshot(`${$testInfo.title}.png`, defaultThresholds);we haven't ran any scenario yet in docker that hadn't had screens.
Hi, well, it's totally ok because it's 2 different systems and they can have slight changes in browsers. I simply run PW docker image locally and generate screenshots there. I also skipped darwin/win32 screenshots to be able to run these tests locally without docker (and don't commit the same files).
Also make sure PW image and a runner in gitlab CI has the same OS version.
Steps:
- Run
docker run --rm --network host -v $(pwd):/work/ -w /work/ --env-file=$(pwd)/test/config/.env.dev -it mcr.microsoft.com/playwright:v1.39.0-jammy /bin/bash - Install dependecies inside docker container
- Run tests with --update-snapshots flag. I'm using this command
"pw:update-snapshots": "yarn pw:run -g snapshot --update-snapshots"and addedsnapshotpostfix to all such tests to easier collect them using--grepflag.
--env-file is optional, of course
Not sure...if you found the solution to this, but I am also currently facing this issue. From what I read online...there is a difference in how graphics are rendered on M1/M2 chips vs Intel chips ....so that causes pixel differences between local and CI. Please lmk if you do find the solution to this.
It is very hard to have the same screenshots if you take them on two different environments. That's one of the reason why I created Argos. All screenshots are taken on CI, then uploaded to Argos (not commited in repo). And then Argos compares it with the good baseline (by using git). It presents several advantages:
- Screenshots are stable (taken on CI)
- Screenshots are not in the repo
- You don't have to run a command locally to update screenshots
Checkout https://argos-ci.com and https://argos-ci.com/playwright if you are interested.
Rayrun is a community for QA engineers. I am constantly looking for new ways to add value to people learning Playwright and other browser automation frameworks. If you have feedback, email luc@ray.run.