Comparing Automated Testing Tools: Cypress, Selenium, Playwright, and Puppeteer

Just want to see the results? Skip to the benchmarks.

Welcome, QA engineers! We'll dive into a detailed comparison of five major automated testing tools: Cypress, Selenium, Playwright, and Puppeteer. Our objective is to understand how these tools perform against one another, dissect their features, and identify their best use-case scenarios.

Background Information

Let's start with a quick refresher on each tool.

Cypress

Cypress is a JavaScript-based end-to-end testing framework. It allows you to write tests that run directly in the browser, making it easier to debug. Here's a quick example of a Cypress test written in TypeScript:

describe('Cypress Test', () => {
  it('Visits the Ray site', () => {
    cy.visit('https://ray.run/')
  })
})

Selenium

Selenium, a pioneer in the testing space, provides a way to automate browsers. It supports a wide range of languages, including JavaScript. Here's how you can set up a basic Selenium test:

import { Builder, By, Key, until } from 'selenium-webdriver';

let driver = new Builder().forBrowser('firefox').build();

driver.get('https://ray.run/');
driver.findElement(By.name('q')).sendKeys('webdriver', Key.RETURN);
driver.wait(until.titleIs('webdriver - Google Search'), 1000);

Puppeteer

Puppeteer, a library developed by Google, provides a high-level API over Chrome DevTools Protocol. It's primarily used for browser automation. Here's a small Puppeteer test:

import puppeteer from 'puppeteer';

const browser = await puppeteer.launch({ headless: "new" });
const page = await browser.newPage();
await page.goto('https://ray.run/');
await page.screenshot({path: 'example.png'});
await browser.close();

Playwright

You're familiar with Playwright, but let's revisit what it brings to the table. As an open-source library developed by Microsoft, it provides a high-level API to control Chrome, Firefox, and Safari browsers. It's reliable and robust for end-to-end testing, and here's how you write a basic test:

import { chromium } from 'playwright';

const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto('https://ray.run/');
await page.screenshot({ path: 'example.png' });
await browser.close();

Playwright Test

Playwright Test, or simply @playwright/test, is a zero-config, TypeScript-based end-to-end testing framework built by the same team as Playwright. It's optimized for both developer productivity and CI/CD environments.

Playwright Test supports all Playwright features including multi-page scenarios, auto-waiting, network interception, and more. It also provides additional features like test parallelization, retries, and reporting.

// Sample test with @playwright/test
import { test, expect } from '@playwright/test';

test('basic test', async ({ page }) => {
  await page.goto('https://ray.run/');
  const title = page.title();
  expect(title).toBe('Welcome to Ray');
});

Use Case Scenarios

While it's important to understand the performance benchmarks of these tools, it's equally vital to consider what each tool is primarily designed for, as this often influences its strengths and weaknesses.

Puppeteer and Playwright, for example, are primarily designed to automate the browser, making them excellent for operations like web scraping or generating screenshots and PDFs of web pages.

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://ray.run/');
  const data = await page.evaluate(() => document.querySelector('body').innerText);
  console.log(data);
  await browser.close();
})();

On the other hand, Cypress, Selenium, and @playwright/test are designed with end-to-end testing in mind. They provide robust capabilities to simulate user interactions, validate application responses, and check the complete flow of your application.

import { test, expect } from '@playwright/test';

test('end-to-end test', async ({ page }) => {
  await page.goto('https://ray.run/');
  await page.click('#login');
  await page.fill('input[name=username]', 'testuser');
  await page.fill('input[name=password]', 'testpass');
  await page.click('#submit');
  const welcomeMsg = await page.$eval('#welcome', el => el.innerText);
  expect(welcomeMsg).toContain('Welcome, testuser');
});

The Testing Environment

Understanding the environment in which these tests are run is critical for interpreting the results accurately. Here are the specifics for our test environment.

Hardware Used

I conducted our tests on a machine with the following specifications:

• OS: macOS 13.2.1
• CPU: (10) arm64 Apple M1 Max
• Memory: 64.00 GB

These tests will give us a good indication of how these tools perform on high-end hardware.

Software and Versions

I used the latest versions of all the testing tools at the time of our testing:

• cypress: 12.16.0
• selenium-webdriver: 4.10.0
• puppeteer: 20.7.3
• playwright: 1.35.1
• @playwright/test: 1.35.1

This ensures that we are comparing the most recent and advanced capabilities each tool has to offer.

Configuration

• Video recording is disabled
• Tracing is disabled

Test Script

I used a consistent script for each tool to ensure a fair comparison. The script performs the following actions:

1. Go to ray.run.
2. Navigate to "Blog" using the link in the header.
3. Pick the "Accessibility Testing with Playwright" article.
4. Scroll into view "Subscribe to newsletter".
5. Fill in first name: test, email: [email protected].
6. Click "Subscribe".
7. Confirm "Now check your email" message.

It looks like this:

Here is an example of how the script was implemented in Cypress:

describe('newsletter subscription', () => {
  it('successfully subscribes', () => {
    // Go to ray.run
    cy.visit('https://ray.run');

    // Navigate to "Blog" using link in the header
    cy.get('a').contains('Blog').click();

    // Pick an article
    cy.get('a').contains('Accessibility Testing with Playwright').click();

    // Scroll into view "Subscribe to newsletter"
    cy.get('div').contains('Subscribe to newsletter').scrollIntoView();

    // Fill in first name and email
    cy.get('input[name="firstName"]').type('test');
    cy.get('input[name="email"]').type('[email protected]');

    // Submit
    cy.get('button').contains('Subscribe').click();

    // Confirm thank you message
    cy.get('p').should('contain', 'Now check your email!');
  });
});

This script uses Cypress's commands to navigate the website, fill in the form, and verify the expected result.

Below is the Selenium WebDriver implementation of the test script in TypeScript:

import { Builder, By, until } from 'selenium-webdriver';

(async () => {
  let driver = await new Builder().forBrowser('chrome').build();

  // Go to ray.run
  await driver.get('https://ray.run');

  // Navigate to "Blog" using link in the header
  await driver.findElement(By.linkText('Blog')).click();

  await driver.wait(until.elementLocated(By.partialLinkText('Accessibility Testing with Playwright')), 1000);

  // Pick an article
  await driver.findElement(By.linkText('Accessibility Testing with Playwright')).click();

  await driver.wait(until.elementLocated(By.xpath("//*[contains(text(), 'Subscribe to newsletter')]")), 1000);

  // Scroll into view "Subscribe to newsletter"
  let element = await driver.findElement(By.xpath("//*[contains(text(), 'Subscribe to newsletter')]"));
  
  await driver.executeScript("arguments[0].scrollIntoView(true);", element);

  // Fill in first name and email
  await driver.findElement(By.name('firstName')).sendKeys('test');
  await driver.findElement(By.name('email')).sendKeys('[email protected]');

  // Submit
  await driver.findElement(By.xpath("//*[contains(text(), 'Subscribe')]")).click();

  // Confirm thank you message
  await driver.wait(until.elementLocated(By.xpath("//*[contains(text(), 'Now check your email!')]")), 10000);

  await driver.quit();
})();

In Selenium WebDriver, navigation and interaction with web elements are accomplished using a range of different locator strategies.

Here's how the test script can be written using Puppeteer:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  // Go to ray.run
  await page.goto('https://ray.run');

  // Navigate to "Blog" using link in the header
  await Promise.all([
    page.waitForNavigation(),
    page.click('::-p-text("Blog")'),
  ]);

  // Pick an article
  await Promise.all([
    page.waitForNavigation(),
    page.click('::-p-text("Accessibility Testing with Playwright")'),
  ]);

  // Scroll into view "Subscribe to newsletter"
  await page.evaluate(() => {
    const element = Array.from(document.querySelectorAll('div')).find(({textContent}) => textContent.includes('Subscribe to newsletter'));
    element.scrollIntoView();
  });

  // Fill in first name and email
  await page.type('input[name="firstName"]', 'test');
  await page.type('input[name="email"]', '[email protected]');

  // Submit
  await page.click('::-p-text("Subscribe")');

  // Confirm thank you message
  await page.waitForSelector('text="Now check your email!"');

  await browser.close();
})();

This Puppeteer script leverages the tool's powerful API to navigate the website, interact with page elements, and validate the resulting messages.

Here's how you would write the test script using Playwright:

import { chromium } from 'playwright';

(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();

  // Go to ray.run
  await page.goto('https://ray.run');

  // Navigate to "Blog" using link in the header
  await Promise.all([
    page.waitForNavigation(),
    page.click('a:has-text("Blog")'),
  ]);

  // Pick an article
  await Promise.all([
    page.waitForNavigation(),
    page.click('a:has-text("Accessibility Testing with Playwright")'),
  ]);

  // Scroll into view "Subscribe to newsletter"
  await page.evaluate(() => {
    const element = Array.from(document.querySelectorAll('div')).find(({textContent}) => textContent.includes('Subscribe to newsletter'));
    element.scrollIntoView();
  });

  // Fill in first name and email
  await page.fill('input[name="firstName"]', 'test');
  await page.fill('input[name="email"]', '[email protected]');

  // Submit
  await page.click('button:has-text("Subscribe")');

  // Confirm thank you message
  await page.waitForSelector('text="Now check your email!"');

  await browser.close();
})();

As you can see, Playwright's API closely mirrors Puppeteer's, given that they are built by the same team. However, Playwright has broader browser support, allowing you to run this test in Chromium, Firefox, and WebKit.

Finally, here is the equivalent script implemented in @playwright/test:

import {test} from '@playwright/test';

test('newsletter subscription', async ({ page }) => {
  // Go to ray.run
  await page.goto('https://ray.run');

  // Navigate to "Blog" using link in the header
  await page.locator('a:has-text("Blog")').click();

  // Pick an article
  await page.locator('a:has-text("Accessibility Testing with Playwright")').click();

  // Scroll into view "Subscribe to newsletter"
  await page.locator('text="Subscribe to newsletter"').scrollIntoViewIfNeeded();

  // Fill in first name and email
  await page.fill('input[name="firstName"]', 'test');
  await page.fill('input[name="email"]', '[email protected]');

  // Submit
  await page.locator('button:has-text("Subscribe")').click();

  // Confirm thank you message
  await page.waitForSelector('text="Now check your email!"');
});

Each tool was tasked with executing this script 100 times, and the following sections will present their performance results.

Performance Metrics

Performance is a crucial factor to consider when choosing a testing tool. I've measured the execution speed of each tool in executing the test script we've defined earlier. Every tool was tasked with running the script 1, 10 and then 100 times, and the total execution time was recorded.

The reason for running the script multiple times is to account for the time it takes to launch the browser, navigate to the website, and execute the script.

Benchmarks

The table above summarizes the results across three tests. The time durations are represented in minutes and seconds format.

As a reminder, test 1 runs the script once, test 2 runs the script 10 times, and test 3 runs the script 100 times.

Here's a table summarizing the results for Test 1, with execution speeds expressed as percentages relative to the fastest tool:

This table clearly illustrates the relative performance of each tool. @playwright/test had the fastest execution speed, serving as our baseline (100%). The rest of the tools are assessed relative to this.

Cypress was the slowest, taking 350% of the time that @playwright/test took. Selenium and Puppeteer were comparable, taking approximately 166% and 164% of the time, respectively. Playwright was closer to the speed of @playwright/test, at 144% of the time.

Here's the table for Test 2, showing execution speeds as percentages relative to the fastest tool:

In the case of Test 2, Playwright emerged as the fastest tool, hence it is our baseline (100%). Cypress was the slowest, taking 142% of the time that Playwright took. Selenium was not far behind, taking 134% of the time. Puppeteer and @playwright/test were relatively close to Playwright, taking 114% and 106% of the time, respectively.

Finally, here's the table for Test 3, illustrating execution speeds as percentages relative to the fastest tool:

In Test 3, Playwright had the fastest execution speed, making it the baseline (100%). The slowest tool was Selenium, taking 149% of the time that Playwright took. Cypress was slightly faster than Selenium, but still slower than the rest, taking 126% of the time. Puppeteer and @playwright/test were much closer to Playwright, taking 115% and 108% of the time, respectively.

Other Factors

Browser Engine

The browser engine can have a significant impact on the performance of these tools. Cypress, Selenium, and Puppeteer all use the Chromium browser engine, while Playwright uses Chromium, Firefox, and WebKit.

Read more about the differences between Chromium, Firefox, and WebKit.

Parallelization

Parallelization can significantly improve the speed of test execution. @playwright/test supports parallelization out of the box, while Cypress, Selenium and Puppeteer require additional services/configuration.

Performance Analysis

When analyzing the performance across Test 1, Test 2, and Test 3, we see different patterns that tell us something about how each tool behaves under varying loads.

Startup Time

The first test was a single run, which could be heavily influenced by the startup time of each tool. Startup time includes the time it takes to launch the browser, navigate to the initial URL, and load the required libraries. This is why tools like Cypress, which has a higher startup time, didn't perform as well in Test 1. Conversely, @playwright/test, with a shorter startup time, outperformed the others.

Repeated Execution

Test 2 involved running the test script 10 times. Here, Playwright was the fastest, and the relative performance of @playwright/test and Puppeteer improved compared to Test 1. This suggests that these tools are efficient at reusing resources and handling repeated executions in a single session.

Scalability

Test 3 pushed the tools to their limit by running the test script 100 times. Here, the efficiency of managing resources and handling potential memory leaks became more prominent. Playwright showed the best performance, indicating that it's highly optimized for long-running and high-volume tests.

Each of these factors could influence the performance of the tools in different testing scenarios. Depending on your specific needs - such as a small number of long-running tests, a high number of short tests, or anything in between - you might find one tool performs better than the others.

Usability and Additional Factors

Performance isn't the only factor to consider when choosing a testing tool. Usability and additional features play a significant role as well.

Learning Curve

Learning curve can impact the speed of test script development and adoption within a team. Selenium, as an older and well-established tool, has a wealth of tutorials, guides, and StackOverflow answers, which can ease the learning process. Cypress, Playwright, and Puppeteer have excellent documentation and growing communities, but might be less familiar to teams used to Selenium. @playwright/test, as a relatively new framework, could present a steeper learning curve, though it is directly related to Playwright, so any knowledge there would transfer over.

Developer Experience

Developer experience is another important factor. Cypress shines here with its real-time reloading and time-travel debugging features, making it a great choice for test-driven development. Playwright and Puppeteer offer a more basic, but solid, testing experience with utilities for intercepting network requests and mocking responses. Selenium's developer experience can be more complex due to its different language bindings and the need to manage browser drivers. @playwright/test, however, simplifies this process with built-in test runners, snapshot and trace viewer capabilities.

Browser Support

Browser support is crucial if your users are spread across different browsers. Cypress, Selenium and Playwright support multiple browsers, including Chrome, Firefox, and Safari, making them versatile choices for multi-browser testing. Puppeteer supports Chrome and Firefox. However, @playwright/test inherits the multi-browser support from Playwright, providing the flexibility required for diverse user bases.

Integration with Test Runners and CI/CD

Integration with test runners and CI/CD tools is important for implementing automated testing in your development pipeline. All these tools integrate well with popular test runners like Jest, Mocha, and others. They also support major CI/CD platforms. However, Cypress stands out with its Dashboard service, providing test recording and parallelization features. @playwright/test also offers excellent CI/CD integration, with features like automatic video recording of test runs.

Community and Ecosystem

A large community and a rich ecosystem of plugins can extend the functionality of the tool and provide help when you encounter issues. Selenium, due to its longer presence, has a vast community and many third-party plugins. Cypress, Playwright, and Puppeteer also have growing communities and an increasing number of plugins. @playwright/test, while newer, benefits from its association with Playwright, sharing much of the same community and ecosystem.

Each of these factors could tip the scale in favor of one tool over the others, depending on your specific requirements.

Use Case Scenarios

Each tool shines in different scenarios based on its strengths and design focus. Let's examine some use cases where you might prefer one tool over another.

Extensive Cross-Browser Testing

If your web application needs to support multiple browsers, Selenium and Playwright are excellent choices due to their broad browser support. They cover Chrome, Firefox, Safari, and in Selenium's case, even Internet Explorer. @playwright/test also covers these browsers, giving it the same advantages.

Test-Driven Development

For developers practicing Test-Driven Development (TDD), Cypress stands out with its real-time reloading and time-travel features, which can boost productivity by providing immediate feedback. @playwright/test also offers a good developer experience, with built-in test runners and snapshot and trace viewer capabilities.

Large Test Suites and CI/CD Integration

If you have a large test suite or require advanced CI/CD integration, Cypress and @playwright/test provide excellent features. Cypress's Dashboard service provides insightful analytics, while @playwright/test offers automatic trace and video recording of test runs.

Complex Interactions and Non-Standard Behavior

For web scraping tasks, automation scripts, and testing complex interactions or non-standard behavior, Puppeteer and Playwright excel due to their low-level API that provides granular control over the Chromium browser.

Migration from WebDriver-based Tools

If you're considering migrating from a WebDriver-based tool like Selenium, Playwright could be an ideal choice. It offers a similar feature set and functionality but brings along enhancements like auto-waiting, network interception, and multi-tab/page support.

In each of these scenarios, the chosen tool can make a significant difference in productivity, ease of use, and the effectiveness of your testing efforts.

The Future of Automated Testing

The field of automated testing is continuously evolving, driven by developments in browser technologies, testing methodologies, and the ever-increasing complexity of web applications.

Looking forward, we see several trends that are likely to shape the future of automated testing:

Increase in JavaScript-based Tools

JavaScript's dominance in the web development landscape has translated to the automated testing world as well. We see this with the rise of tools like Cypress, Playwright, Puppeteer, and @playwright/test, which all leverage JavaScript/TypeScript. This trend is likely to continue as more development teams standardize on JavaScript across their tech stacks.

Greater Focus on Developer Experience

As tools compete for adoption, there's an increased focus on developer experience. This includes real-time feedback, rich debugging capabilities, intuitive APIs, and seamless integration with popular development tools.

Shift Towards End-to-End Testing

Unit testing and integration testing remain vital, but there's a growing appreciation for end-to-end testing. Tools like Playwright and Cypress that facilitate easy and robust end-to-end testing are likely to gain even more popularity.

Cloud-based Testing

Cloud-based testing allows for scalable, parallel test execution and reduces the need for maintaining local testing infrastructure. While Selenium has been offering this through third-party services like Sauce Labs and BrowserStack, Cypress and @playwright/test offer their own solutions.

AI and Machine Learning

AI and Machine Learning can bring about a revolution in test case generation, anomaly detection, visual testing, and flaky test identification. This is a new frontier, but we're already seeing some movement in this direction. We are even seeing AI assisting in writing test scripts and answering developer questions (see Ask AI).

The future of automated testing is exciting, and it's important to stay informed about these trends when choosing your testing tools.

Conclusion

Automated testing has become an integral part of modern web development, and the choice of tools can significantly impact your testing effectiveness and productivity. Cypress, Selenium, Playwright, Puppeteer, and @playwright/test each offer unique strengths and could be the best choice depending on your specific requirements.

Our performance benchmark revealed that @playwright/test and Playwright tend to outperform the others in terms of raw speed, particularly for longer and repeated test runs. However, when it comes to usability, developer experience, community support, and additional factors, the right choice can vary.

For instance, Selenium's wide browser support and large community might make it the best choice for some. At the same time, Cypress's developer experience and built-in dashboard could tip the scale in its favor for others. Puppeteer and Playwright provide excellent low-level control, making them suitable for complex interactions, while @playwright/test combines this with efficient performance and good CI/CD integration.

In short, there's no one-size-fits-all answer. The best tool depends on your team's needs, expertise, and the specific challenges of your project. I hope that this article has provided you with a comprehensive understanding of these tools' performance characteristics and their pros and cons, helping you make an informed decision.