Flaky tests are ending the main process

Hi, you are giving very little information to go by. Are you using @playwright/test in NodeJS (JavaScript/TypeScript)? If not, what else?

What kind of process is that release process? Some CI pipeline? What CI system are you using? How does the pipeline definition look?

Hi,

Thank you for responding!

I'm using Playwright in Node.js with JavaScript.

The release process is just an example.

Another example would be having 2 Playwright projects under 2 different projects in a monorepo, which should run one after the other using an npm script from the root of the monorepo.

If the first test run has a flaky test, it won't proceed to the second test run.

The following is my reporter setup in playwright config:

reporter: [['list', { printSteps: false }], ['html']],

I don't know what the npx playwright test exit code is when there are flaky tests. But whatever it is, if the calling process needs to continue to the next step, then the calling process needs to deal with it.

Any test run that doesn't pass 100% will exit with non-zero. I'm inferring that the concern is in a pipeline, so you would either A) fix the flake, B) skip the flaky tests until you can do A, or C) configure the runner to allow the PW test stage to fail without killing the pipeline.