How to continue on failure?

See the maxFailures values 😉 https://playwright.dev/docs/next/test-parallel#limit-failures-and-fail-fast

You need to use page fixture in each test block so that every test is isolated, this could result in running whole test suite even if one test fails

maybe try using expect.soft()

You can use the page fixture something like this: https://github.com/LambdaTest/playwright-sample/blob/7c529515527f711ad3864f1753d0ec73c819a52c/playwright-test-js/lambdatest-setup.js#L41-L71

You will need to create the new browser and page object from here and import it into your tests. This code is for creating the page object for running on LambdaTest. You can use browserType.launch() method to run in local.

Else, set maxFailures: https://playwright.dev/docs/next/test-parallel#limit-failures-and-fail-fast if that solves your issue.

I'm a little confused here. Currently I have 16 tests across 5 files. Each file has its own page object and fixture. Each test block uses its corresponding page fixture. The page fixture runs through the login process, so each test block includes a fresh login. I don't know how to make them more isolated than that. Yet if any one of the tests fails, the rest are skipped.

We're using one worker, but the order of the tests doesn't matter. So if the second test that's run fails, the results of the suite are passed: 1 failed: 1 skipped: 14

Change all the expect() to expect.soft()... Have you tried deadbeef's suggestion?

Are you running in serial mode? You need to use parallel mode if you want the runner to continue running tests after a failure.

If you don't specify a mode, it should be parallel. https://playwright.dev/docs/next/test-parallel#serial-mode

There are 3 modes:

mode "default"|"parallel"|"serial" (optional) https://playwright.dev/docs/api/class-test#test-describe-configure

The default is to run tests in a single spec sequentially, but to run different specs in parallel, depending on number of workers. (Probably when testConfig.fullyParallel is enabled, the default behaviour is changed) https://playwright.dev/docs/api/class-testconfig#test-config-fully-parallel

"parallel" will even run tests from the same spec in parallel.

"serial" is sequential but will skip all subsequent tests in the same spec if the previous one fails.

I suspect "serial" is the mode you are using.

I don't have a mode configured. I'll look into that, thanks.

expect.soft feels like the wrong solution, but I'll give that a go if modes don't change anything for me.

Specifying fulyParallel: true changed nothing. I'll look into soft assertions, but I thought that was for continuing in a test block, not continuing a whole suite.

Same results with expect.soft(). 1 passed, 1 failed, the rest skipped.

Hi, can you share your playwright.config.js/ts and the 'outline' of one of your specs, to see what pattern the specs are following?

Thanks, everyone. I'm trying to create a reproduction that people can look at. It's at least possible it's an issue with the @mands/nx-playwright plugin. I'll update if I come up with something.

As much as i prefer using pnpm, saw there are issue using PW and pnpm. Think there was a comment in the last day or two explaining the issue., and doesn't look like that will be fixed anytime soon.

Hi friends. I created a reproduction. I think what it comes down to is that I was using fixtures wrong. I'd love it if someone would take a look and say, "yeah that's wrong" or "no that should work." If nothing else, it's there for future generations. Basically I assume I wasn't supposed to use the same login page in every fixture. https://github.com/cdcasey/skip-test-repro

Hi, your usage of fixtures looks fine to me. Every test uses a new instance of the pages instead of re-using one, so that's good for keeping tests isolated, it's what Playwright advocates.

I don't know what's causing the symptom you have (that one failing test making all the other ones ignored). Probably good to create a GitHub issue for it.

One small (probably unrelated) thing I noticed which could cause a problem: https://github.com/cdcasey/skip-test-repro/blob/main/tests/login/login.page.ts#L12

await this.page.goto('/');
  await this.page.reload();
  await this.loginButton.click();
}```

Do you really need the reload for anything?

I think it could cause a net_aborted error, because maybe it goes like this:
1. login page is already displayed
2. page reload is initiated
3. login button is clicked for the 'not yet re-rendered page' (a button matching the locator was already there on the page before the reload/re-render)
4. the click itself causes a form post, and Chrome/Chromium will abort the reload network request

Not sure if that happens in your case, but we did have a net_aborted in a similar situation.

Good callout. I'm not using it in my actual code. For some reason I assumed I needed it because I was simulating the login with state, but that was erroneous.