Stress testing Playwright tests using GitHub Actions

Lately I've been spending a lot of time batling flaky tests and I came across this article on Stress testing Playwright tests using GitHub Actions. I thought it was a great article and wanted to share it with you.

Stress testing Playwright tests using GitHub Actions

To ensure the highest quality and reliability in your continuous integration and continuous deployment (CI/CD) pipeline, it's crucial to rigorously validate the robustness of your tests. Despite best efforts, flaky tests can occasionally infiltrate the pipeline, leading to frustrating and costly delays.

A proven strategy to expose potential failures is to execute tests multiple times. This approach is particularly effective when using Playwright, a powerful testing tool. For instance, to repeat a specific test 20 times, you can execute the following command:

npx playwright test {{insert test name here}} --repeat-each=20

However, tests may behave unpredictably in different environments, especially within the pipeline. It's a common scenario where a test, seemingly flawless when merged, starts failing during pipeline execution.

To mitigate this, those utilizing GitHub Actions can establish a custom workflow. This workflow enables the repeated execution of the new test within the pipeline across various environments, providing a more comprehensive assessment of the test's robustness.

Setting up a custom workflow

Here's how to set up this workflow in GitHub Actions:

name: Playwright Tests
on:
  workflow_dispatch:
    inputs:
        repeat-each:
          default: 5
          type: number
          description: The number of times to repeat the test
        test-path:
          required: true
          type: string
          description: A relative path to the test file under test
jobs:
  test:
    timeout-minutes: 60
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - uses: actions/setup-node@v3
      with:
        node-version: 21
    - name: Install dependencies
      run: npm ci
    - name: Install Playwright Browsers
      run: npx playwright install --with-deps
    - name: Run Playwright tests
      run: npx playwright test "${{ github.event.inputs.test-path }}" --workers=1 --repeat-each=${{ inputs.repeat-each }}
    - uses: actions/upload-artifact@v3
      if: always()
      with:
        name: playwright-report
        path: playwright-report/
        retention-days: 30

Running the Workflow

Now go to the Actions tab on your repository, select "Playwright Tests", and click on the Run workflow button.

This will immediately trigger the workflow, prompting you to enter the relative path to the test file you wish to test and the number of times you wish to repeat the test. If the test passes, you'll know it is working as intended.

Common Flakiness Causes

If you are venturing into the world of Playwright testing, I highly recommend reading Mastering the Art of Detecting and Handling Flaky Tests in Playwright to avoid common pitfalls.

Implementing such a workflow elevates the reliability of your test suite, ensuring that tests are rigorously vetted before being integrated into the main pipeline. This not only enhances the quality of your software but also significantly reduces the likelihood of pipeline bottlenecks caused by flaky tests.