Answer

Best Practices for Configuring Automated Tests with CI/CD

To minimize resource usage and costs, you can follow several best practices.

Run Tests Frequently

Ideally, run your tests on each commit and pull request. Playwright comes with a GitHub actions workflow so that tests will run on CI for you with no setup required. You can also set up Playwright on the CI environment of your choice.

Use Linux on CI

Linux is cheaper, so it's advisable to use it on the Continuous Integration (CI) platform.

Lint Your Tests

Linting helps catch errors early. Use the @typescript-eslint/no-floating-promises ESLint rule to ensure there are no missing awaits before the asynchronous calls to the Playwright API.

Run Tests in Parallel

Playwright runs tests in parallel by default, saving time and resources. If you have many independent tests in a single file, run them in parallel using test.describe.configure({ mode: 'parallel' });.

Use Sharding

Sharding allows a test suite to be executed across multiple machines or processes concurrently, reducing overall execution time significantly.

npx playwright test --shard=1/3

Set Up CI

Ensure CI agent can run browsers, install Playwright, and run your tests.

npm ci 
npx playwright install --with-deps
npx playwright test

Set Workers to "1" in CI

For stability and reproducibility, set workers to "1" in CI environments. This ensures each test gets the full system resources, avoiding potential conflicts.

import { defineConfig, devices } from '@playwright/test'; 
export default defineConfig({ 
  workers: process.env.CI ? 1 : undefined,
});

Use Sharding for Wider Parallelization

Distribute tests across multiple CI jobs using the --shard option in Playwright command line tool.

npx playwright test --project=${{ matrix.project }} --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}

These practices will help you configure a large number of automated tests with CI/CD while minimizing resource usage and costs. For more tips on efficient Playwright test scripts, check out this blog post.