Stress testing Playwright tests using GitHub Actions

Discover how to boost your CI/CD pipeline's reliability by implementing robust Playwright tests within GitHub Actions. Learn to create custom workflows for repeated test executions, ensuring software quality and reducing pipeline delays.
  1. Stress testing Playwright tests using GitHub Actions
    1. Setting up a custom workflow
      1. Running the Workflow
        1. Common Flakiness Causes

          Lately I've been spending a lot of time batling flaky tests and I came across this article on Stress testing Playwright tests using GitHub Actions. I thought it was a great article and wanted to share it with you.

          Stress testing Playwright tests using GitHub Actions

          To ensure the highest quality and reliability in your continuous integration and continuous deployment (CI/CD) pipeline, it's crucial to rigorously validate the robustness of your tests. Despite best efforts, flaky tests can occasionally infiltrate the pipeline, leading to frustrating and costly delays.

          A proven strategy to expose potential failures is to execute tests multiple times. This approach is particularly effective when using Playwright, a powerful testing tool. For instance, to repeat a specific test 20 times, you can execute the following command:

          npx playwright test {{insert test name here}} --repeat-each=20

          However, tests may behave unpredictably in different environments, especially within the pipeline. It's a common scenario where a test, seemingly flawless when merged, starts failing during pipeline execution.

          To mitigate this, those utilizing GitHub Actions can establish a custom workflow. This workflow enables the repeated execution of the new test within the pipeline across various environments, providing a more comprehensive assessment of the test's robustness.

          Setting up a custom workflow

          Here's how to set up this workflow in GitHub Actions:

          name: Playwright Tests
                    default: 5
                    type: number
                    description: The number of times to repeat the test
                    required: true
                    type: string
                    description: A relative path to the test file under test
              timeout-minutes: 60
              runs-on: ubuntu-latest
              - uses: actions/checkout@v3
              - uses: actions/setup-node@v3
                  node-version: 21
              - name: Install dependencies
                run: npm ci
              - name: Install Playwright Browsers
                run: npx playwright install --with-deps
              - name: Run Playwright tests
                run: npx playwright test "${{ github.event.inputs.test-path }}" --workers=1 --repeat-each=${{ inputs.repeat-each }}
              - uses: actions/upload-artifact@v3
                if: always()
                  name: playwright-report
                  path: playwright-report/
                  retention-days: 30

          Running the Workflow

          Now go to the Actions tab on your repository, select "Playwright Tests", and click on the Run workflow button.

          Running the workflow

          This will immediately trigger the workflow, prompting you to enter the relative path to the test file you wish to test and the number of times you wish to repeat the test. If the test passes, you'll know it is working as intended.

          Common Flakiness Causes

          If you are venturing into the world of Playwright testing, I highly recommend reading Mastering the Art of Detecting and Handling Flaky Tests in Playwright to avoid common pitfalls.

          Implementing such a workflow elevates the reliability of your test suite, ensuring that tests are rigorously vetted before being integrated into the main pipeline. This not only enhances the quality of your software but also significantly reduces the likelihood of pipeline bottlenecks caused by flaky tests.

          Thank you!
          Was this helpful?

          Now check your email!

          To complete the signup, click the confirmation link in your inbox.

          Subscribe to newsletter

          You will receive a weekly email with the latest posts and QA jobs.

          AboutQuestionsDiscord ForumBrowser ExtensionTagsQA Jobs

          Rayrun is a community for QA engineers. I am constantly looking for new ways to add value to people learning Playwright and other browser automation frameworks. If you have feedback, email luc@ray.run.