Playwright tests inside project or should playwright tests be it’s own repo?
I have been trying to find resources/information whether to have the playwright E2E tests within the project repo or should the playwright tests be it’s separate repo? Does it get complicated to maintain the playwright tests in the project repo as the devs are working on development?
Are there any good articles/resources I can read to help with making a decision? Any example repos on GitHub I can take a look at?
Are there factors that devs and testers should consider? Pros and cons.
Any guidance or advice would be appreciated. Thank you!
This thread is trying to answer question "Should Playwright E2E tests be kept within the project repository or in a separate repository?"
Keeping tests in the same repo has a lot of pros such as the CI/CD pipeline integration is a lot simpler, and you're tests will/should always be in sync with the application under test version.
In my role, we currently have our ui tests in their own repo (and api tests in their own repo), we solved the CI/CD problem in github action with workflow dispatch jobs that kick off other repo runs. It's working for us but adds a lot of complexity.
The reason we decided not to keep our code in the main application repo was because our strict PCI compliance, every P/R must be reviewed by a separate developer for our production codebase (we deal with raw cc numbers). We are a small team with 8 devs and 1 dedicated tester (and me manager). So this allows us to move quickly in our automation repo but we do have issues when we update a test due to a feature branch, but that feature branch isn't pushed into our develop/staging environment where our tests run on every build, so we have to create p/rs and link them to our application code p/rs (which is also not great, but a tradeoff we are willing to work with).
Thank you for the information. Appreciate it.
What does the work flow look like when tests need to be updated due to changes/updates. Do the playwright tests get updated when the dev tickets are in QA? Are the tests updated after the release is deployed and the current changes are checked manually during the regression process?
QA shouldn't be a separate step in your process. You should write the tests alongside the SWE. In our org, the SWE writes the tests. No code should make it to your main/production branch without having tests. It shouldn't even be in code review without having tests.
I've never seen a place where everything is tested before it is released though. So, tests can/should be updated independent of the application. They still live with the code, but they are constantly being improved independent of features.
One more thing to consider is that Playwright can be used not only for end-to-end tests, but also for component testing.
The tests for detailed bells and whistles of the frontend could be written as component tests (probably by the developers) and/or as isolated frontend tests (integrated frontend, but with the whole backend mocked).
Then you can write fewer end-to-end tests, just for the business critical flows and leave the details to tests in the lower levels of the test pyramid.
Such component tests and isolated frontend tests definitely belong in the same repo as the app. It makes it easy for devs to run them as part of their 'inner loop', even before pushing their commits to CI.
We keep component and functional testing with the code repository. We do have a small E2E regression suite of tests that are agnostic to our components, and we use this suite as our integration verification in our staging environment. We have multiple pipelines delivering to production, so we needed a way to ensure that all component pipelines have been verified to our critical business workflows. This regression suite is less than 20 tests, so it is not overly painful to maintain.
Some other considerations that we faced was that our teams were performing experiments and pilots that were driven by feature flags. This was becoming a nightmare to maintain in an centralized repository and would cause extensive time investigating flaky tests based on feature flag mis-configuration. Having that logic mocked at the component level in the code repository made more sense for us. The external E2E solution only has to deal with the deployment target settings not a future state that might be implemented.
Hi, I have to admit we ourselves are not using component testing or isolated frontend tests yet. It is on the wishlist to get a better balance in the test automation strategy, and to avoid slow full end-to-end tests, where state is hard to arrange, for frontend details. So I don't know as much about it as I'd like. I have to refer you to https://playwright.dev/docs/test-components and https://playwright.dev/docs/network#handle-requests As far as I can see it only requires Playwright, as long as your UI framework is supported for Playwright's (still experimental) component testing.
Rayrun is a community for QA engineers. I am constantly looking for new ways to add value to people learning Playwright and other browser automation frameworks. If you have feedback, email luc@ray.run.