Recommended parallel setting for Azure Devops?

I also notice if I cut the test suit in half they will pass.... so it seems that the more tests the more demanding on cpu?

I would not configure the number of workers, leave it out of the config, unless you have a specific reason to deviate from the default (recommended) value.

https://playwright.dev/docs/api/class-testconfig#test-config-workers "Defaults to half of the number of logical CPU cores. Learn more about parallelism and sharding with Playwright Test."

Ideally slowMo should not be used. It's slowing down tests, but still not guaranteeing that flakiness is solved.

Using the best practices, most things should already wait the minimum required time (auto-waiting locator actions and auto-waiting assertions): https://playwright.dev/docs/best-practices

Record traces (retain-on-failure) and publish them from CI to some static web hosting like GitHub Pages or Azure Blob Storage web container, or as artifacts, so you can easily troubleshoot tests failing in CI. https://playwright.dev/docs/trace-viewer-intro

Thanks. I have found if I remove the sloMo then all tests fail both local and remote. Currently, sloMo 1000 gets more tests passing in the pipeline

slowMo 1000 is like a hammer. Same counts for waitForTimeout. If you care about having a combination of speed and reliability, it's better to invest in finding the root cause of tests failing (the traces plus trace viewer can help with that).

Are you using Locators for everything? (You should) https://playwright.dev/docs/locators

Are you avoiding force: true? (Force discards the auto-waiting, which is there to make tests both reliable and optimally fast)

Be sure to read the best practices and to apply them. That will be a solid foundation. But: Even when using the best practices, your app may have issues with fast users. Many apps don't block interactions when they are in initialising or processing state, but they should. Ideally you/the devs fix the application.

If fixing the app is no option, you have to analyse and try to find 'signals' to detect when the app is ready for a next step. Add intelligent waits for that. Like wait for a network response which is always received last (you can use waitForResponse), or a page which finishes initialisation by setting focus to a field (you can use await page.locator('#userName:focus').waitFor() for such case. Or some element which appears when the page is fully loaded (await page.locator('#logOut').waitFor()), etc.

Chrome DevTools helps investigate these things. You can enable network throttling with 'Slow 3G' to both reproduce the app's issues with fast users, and to see which things are changing on screen last or network responses always received last (may be used as the 'signal' for app readiness).

(But the best thing is still fixing the app)

Yes, I am using the locators so waiting should be automatic. Also, the test failures are random on the pipeline. A test may fail on one pipeline run, but pass on the next.... random.

I would not recommend more than 2 workers… i also had a similar issue later i found the problem was with the available memory and cpu usage…

I removed the workers to get the default but still same issue.

@vipinphogat did you run it with the azure pipeline self hosted linux agent below? This one gives me 2 cpus and about 3 gigs free ram

pool:
        vmImage: "ubuntu-20.04"

Nope… mine is a windows based agent …irrespective of OS please note that parallel execution requires more memory and cpu power… i would suggest you to increase the memory and then try multiple workers

Hi, just repeating, as described earlier, even with locators and Playwright's auto-waiting there are many cases where you have to either fix the app (to block fast users (like Playwright) from interacting while the app is not 'ready'), or to put in 'smart' waits like await page.waitForResponse or await locator.waitFor(). (For more explanation see earlier message.)

Finding out which places require such additional 'smart' waits is a serious challenging engineering task. But it's the only way to make your tests both fast and reliable.

The trace files make it easier to do that analysis but still quite challenging. It's often good to collaborate with the app developers on this.

@vipinphogat @refactoreric do you typically run your front end app in the Azure pipelie to test against? Or do you test against the front end hosted else where?

We have our application (frontend and various backend services, complex heterogeneous architecture) deployed somewhere else.

I found the solution, I switched from ubuntu agent to macOS-13 which had more cpu and ram. This allowed my tests to pass 100% along with the app being hosted in the pipeline during the pipeline run

@mxschmitt maybe this should be mentioned in the Playwright CI documentation since it is in the context of Azure Devops which is owned by MS?

Great 👍😊…. So CPU and available memory was the problem….

depends highly on the tests, and why they were failing. In general they should pass on every OS if they were written reliably and follow our best practises.

The ultimate test would be to not run any tests parallel on the ubuntu agent and see if the tests are failing

The tests failed on Ubuntu agent without running parallel

I think there is a misunderstanding... It wasn't because of the OS itself. Azure Devops uses 2 cpus and 7 gigs of ram on all their self hosted agents, with the exception of the mac os agents which are 3 cpus and 14 gigs of ram.

I had 37 tests for a Angular app that was nothing out of the ordinary. That was not enough for the Azure self hosted agents that have 2 cpus. I think that it is safe to say that it will be most people's experience if they use Azure Devops for Playwright on a typical app and test suite size. It would be nice to put that in Playwright documentation because I really had to dig to find that and I lost like 3 days trying different things to get them to pass on Azure.

ah, following playwright best practises means that for two logical cpu cores you should have one worker. In most cases for CI runners we recommend workers: 1, see here: https://playwright.dev/docs/ci#workers how many workers did you have configured?

I first tried 2 workers to no success, then I tried 1 worker to no success, then I just removed the workers entirely to let the default run. None of those worked for me on the 2 cpu.

To note, this is with the app being hosted in the azure pipeline also using localhost. I used http-server instead of ng because it is lighter. This is making use of playwrights web server in azure pipeline.

and what were the errors? We need to know how the errors looked like, how your test look like etc. to be able to reason more about it. 🙂