Rayrun
← Back to Discord Forum

How to make test execution time not block CI/CD pipeline

Ashish Vishnoiposted in #help-playwright
Open in Discord
Ashish Vishnoi
Ashish Vishnoi

I have 100+ tests automated using playwright, but so far I was running in local and now planning to have them as part of CI/CD pipeline. But there is a concern on the test execution time, in parallel run also it takes around 2 hrs to complete those tests. How to configure it with CI/CD and how to use less and less resources remotely to avoid huge costs to run them. <#1054804523652231198>

This thread is trying to answer question "How to configure automated tests with CI/CD and use less resources remotely to avoid high costs and long execution times?"

4 replies
insane_rookstar
insane_rookstar

100 shards with --workers=100%? xD

insane_rookstar
insane_rookstar

Do you have your own CI or use Azure (or similar)?

insane_rookstar
insane_rookstar

We have 150 tests (3 viewports, so 450 in total) and it takes around 30min to complete the pipeline. We use 5 shards with 8-12 workers each

What parallelism do you use? 2 hours is abnormally long.

Answer

Best Practices for Configuring Automated Tests with CI/CD

To minimize resource usage and costs, you can follow several best practices.

Run Tests Frequently

Ideally, run your tests on each commit and pull request. Playwright comes with a GitHub actions workflow so that tests will run on CI for you with no setup required. You can also set up Playwright on the CI environment of your choice.

Use Linux on CI

Linux is cheaper, so it's advisable to use it on the Continuous Integration (CI) platform.

Lint Your Tests

Linting helps catch errors early. Use the @typescript-eslint/no-floating-promises ESLint rule to ensure there are no missing awaits before the asynchronous calls to the Playwright API.

Run Tests in Parallel

Playwright runs tests in parallel by default, saving time and resources. If you have many independent tests in a single file, run them in parallel using test.describe.configure({ mode: 'parallel' });.

Use Sharding

Sharding allows a test suite to be executed across multiple machines or processes concurrently, reducing overall execution time significantly.

npx playwright test --shard=1/3

Set Up CI

Ensure CI agent can run browsers, install Playwright, and run your tests.

npm ci 
npx playwright install --with-deps
npx playwright test

Set Workers to "1" in CI

For stability and reproducibility, set workers to "1" in CI environments. This ensures each test gets the full system resources, avoiding potential conflicts.

import { defineConfig, devices } from '@playwright/test'; 
export default defineConfig({ 
  workers: process.env.CI ? 1 : undefined,
});

Use Sharding for Wider Parallelization

Distribute tests across multiple CI jobs using the --shard option in Playwright command line tool.

npx playwright test --project=${{ matrix.project }} --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}

These practices will help you configure a large number of automated tests with CI/CD while minimizing resource usage and costs. For more tips on efficient Playwright test scripts, check out this blog post.

Related Discord Threads

TwitterGitHubLinkedIn
AboutQuestionsDiscord ForumBrowser ExtensionTagsQA Jobs

Rayrun is a community for QA engineers. I am constantly looking for new ways to add value to people learning Playwright and other browser automation frameworks. If you have feedback, email [email protected].