Can anyone help me figure out what things to consider looking into when their tests fail in the CI with timeout of 30s for page to load as page was closed. But when running it locally it works just fine. I initially assumed it might be difference of workers since locally it was using 6 workers and CI was using 1 worker but then I used the --workers flag to set them all to 4 and that improved some of the errors in CI. I jumped it to 8 workers and now. I've done a lot of other debuggin as well and am unable to pinpoint the issue here. Would appreciate any help with this. Thank you
P.S. I only have about 80 tests. Majority of those are unit tests which are all passing even on the CI. There is 2 e2e tests which are failing. Sometimes one passes sometimes it fails in the CI only. It is really odd.
This thread is trying to answer question "Why do Playwright tests fail in the CI environment with a timeout error but pass when run locally, and how can this issue be resolved?"
Was seeing the same if not more errors when it was running with only 1 worker on the CI pipeline, when I increased it to 8 it showed less errors but still erroring 30000ms timeout as Page closed
Generally CI machines a usually minimal configs, running more then 1 or 2 may be problematic, dictated more by the cores on the box or total thread, 2 core iwth 2 thread give 4 possible, and one would be max. Slog through the docs PW will suggest no more then 25% of your possible threads should be used... One might get away with using 50% or 75% if you can get away with it, but not likely. In the 1.39 release video or was it the PW Testing Services in videos they kind of touch on and showed running with toooooooo many workers. Start with 25% as per the convention then step up.
Hi, the others are right that more workers makes it more tricky. Each worker will be more resource constrained.
Don't know why it seemed different for you in practice.
Anyway if tests fail in CI, using the trace files and trace viewer are the best way to debug: https://playwright.dev/docs/trace-viewer-intro
I prefer setting trace to 'retain-on-failure', but if you have retries configured, you can choose 'on-first-retry'.
That is interesting, so i tried lowering to 1 worker again but i still see the same thing, all unit tests pass and the e2e tests fail saying Test timeout of 30000ms exceeded. Error: locator.innerHTML: Page closed Error: locator.textContent: Page closed etc....
Yes unit tests that i created always pass. The 2 e2e tests which i built for implementing e2e tests the first two being for authentication are failing on CI but not on local
Simply standing up a machine and running tests on it, really need to ensure the machine has enough memory, cpus and what ever is needed. Especially if the only problem you can can say at this point is "Works on my machine but not in CI" Would be exactly the issue if there isn't enough memory or cpu... Even better on your local machine, pop out a memory chip or to or find a way to get you machine to mirror the CI machine.
I tried that and all the unit tests remain passing but the e2e tests didn't. So i tried to change the
timeout to 0 as well since it was still timing out after
120000 and then i saw that 1 part on e2e passed(setup project which helps setup the authentication). However, none of the other e2e tests were even listed which is weird. The CI ran for 2 hours and nothing happened to the other 2 e2e tests
Once i changed the timout to 0, the errors dissapeared. So i don't think it was a Target Closed issue but the browser closed because the time ran out. However, now the tests just don't run on CI. P.S. I have changed to 1 worker as you advised on both local and CI
Rayrun is a community for QA engineers. I am constantly looking for new ways to add value to people learning Playwright and other browser automation frameworks. If you have feedback, email email@example.com.