Detecting Flaky Tests in Playwright with Havoc

Introduction

Havoc is crafted for developers using Playwright. Its primary function is to detect flaky tests in your Playwright test suite, a common challenge in modern web development.

If you have not already, read my earlier article about what causes flaky tests and how to prevent them.

Simplicity Behind Havoc

Havoc operates on a straightforward principle: it intentionally throttles network responses to induce flakiness. This method helps in identifying tests that might fail under varied network conditions. Here's a peek into its simplicity:

import { Page } from "@playwright/test";
import { setTimeout } from "timers/promises";

// Generate a random number between min and max
const random = (min, max) => Math.floor(Math.random() * (max - min + 1)) + min;

export const havoc = async (page: Page) => {
  await page.route("**/*", async (route) => {
    await setTimeout(random(500, 1000));
    await route.continue();
  });
};

You could just copy the above snippet to your codebase, or you could use the npm package to simplify the process. The andvantage of the NPM package is that as the methods to induce flakiness evolve, so will Havoc.

Setting Up

1. Install Dependency:

   $ npm install playwright-havoc -D

2. Import and Utilize:

   import { test } from "@playwright/test";
   import { havoc } from "playwright-havoc";
   
   test.beforeEach(async ({ page }) => {
     await havoc(page);
   });

Workflow

Just add await havoc(page) to the specific tests that you suspect to be flaky and run your test suite. Havoc will automatically throttle network responses to induce flakiness.

You can also add await havoc(page) to the beforeEach hook to induce flakiness in all tests.

Caution

Havoc is intended solely for troubleshooting. It should not be deployed in your CI/CD pipeline, as it substantially prolongs test durations.

Not a Silver Bullet

Havoc is not a silver bullet. It is a tool that helps you identify flaky tests that are flaky as a result of network related race conditions. However, it does not handle scenarios such as: out of order execution or flakiness as a result of unexpected side-effects (like a modal closing unexpectedly). I am still experimenting with ways to surface those scenarios. Your input would be appreciated.

Share Your Feedback

Please submit examples of flaky test scenarios not surfaced by Havoc. This will help me to improve the tool and make it more robust.