Performance Testing in Playwright?

Maybe if there is possiblity to use CDP for network or lighthouse pkgs

I think you can look into k6 or artillery. I have used k6 and it is a great tool for performance testing. What is more have a look at it. As this mighty https://k6.io/blog/get-started-with-k6-browser/#writing-the-test

Also if you could give more info what you are trying to do would be nice and easier to give you some help in the topic 🙂

I would say the author mean web performance

This can be used as an inspiration https://www.youtube.com/watch?v=IrK-XDH72bw Didn't have time to try myself or play with it, but... looks like it's going to work

yes, this is great video, he made repo for JS https://github.com/unlikelyzero/js-perf-toolkit

Website Browser performance I would say yes (google lighthouse metrics) For running load or stress testing playwright wouldn’t scale well.

You could use playwright to generate .har files easily and send those for another load test tool.

https://github.com/artilleryio/artillery/tree/main/examples/browser-load-testing-playwright

https://www.artillery.io/blog/load-testing-with-real-browsers

So we are currently looking at doing some load testing on our platform and have researched on some tools such as k6 and artillery. In this context, in talking about web performance mainly

We currently use playwright for all our tests so thought it might be worth also doing performance testing using playwright also

👋 member of team Artillery here. Playwright + Artillery scale pretty darn well for load testing, certainly better than anything else out there as we let you use AWS directly! What sort of web performance tests are you looking to set up @doublez? something like tracking Core Web Vitals on certain pages over time?

We're currently still in some planning stages of what we want to do precisely. We will probably be focusing on checking the performance of our graphql queries, tracking some web vitals and using it to see whether or not we need to add more resources on the platform

I have a following scenario. Do you guys think it make sense to use k6 or artillery for it, or what approach do you recommend? (We are using Playwright for frontend E2E tests so it would make sense for me to reuse them as much as possible)

The problem: Some clients are complaining that our app takes to much time to load. Devs want to do some upgrade. Unfortunatelly as a QA I don't have any metrics how much time it takes for the app to load and users can start do their thing. And I want to get something in my hand to evaluate the upgrade. I don't think it is necesary to do load testing in a form of opening the app thousands times in parallel. I would be happy to do it hundred times in serial with my local machine. But I need to get some meaningful results and metrics. I don't think it make sense just to record test result run time, because it can be pretty random as it would also account for browser start, test setup and teardown...

@darill This can be done with Artillery. We have an example here that shows how to track metrics that are specific to the app you're testing (so you would only measure in-app actions, and not browser startup time etc): https://github.com/artilleryio/artillery/tree/main/examples/browser-load-testing-playwright#example-3-tracking-custom-metrics-for-part-of-the-flow

You can also configure Artillery to run one instance of Playwright at a time, and create a new one as soon as the previous one finishes with a phases configuration that looks like this:

• duration: 60 # create new browsers for 60 seconds arrivalRate: 1 # create no more than one new browser per second maxVusers: 1 # only allow a max of one active browser at a time

(Docs: https://www.artillery.io/docs/guides/guides/test-script-reference#phases---load-phases)

hi @hveldstra , I am considering using artillery on a new project that has come up. I need to saturate an ec2 instance to know how many users we can support implementing a series of steps to get an initial baseline. https://github.com/artilleryio/artillery/tree/main/examples/browser-load-testing-playwright what is the median cost to run this on fargate in your production workloads?

I found https://www.artillery.io/blog/this-load-test-cost-us-how-much. However this is aws lamda and docs does not list playwright in supported engines. If we ran the same 15 minutes on fargate what would be the cost?

Fargate is very cheap too @leblancmeneses. Artillery uses 4 vCPU/8GB RAM for each worker by default. Taking us-east-1 pricing it'll cost:

0.04048 * 4 + 0.004445 * 8 = $0.19748 / hour per worker

How much load each worker can generate depends on your test scripts. For simplicity's sake let's say we can run one Chrome instance for each vCPU. That would mean that each Artillery worker could run up to 4 concurrent browsers at a time. With 100 workers you could run up to 400 concurrent virtual users. Let's say each VU session lasts 3 minutes. That's means that 100 Artillery workers, running 4 VUs concurrently each for an hour would give you: 60/3* 4 * 100 = 8k total virtual users in one hour, which would cost $19.7 in Fargate fees.

You can also customize the number of vCPUs per worker on Fargate up to 16, which would let you run more browsers concurrently (at higher cost).

There's more to total cost than just direct Fargate costs, e.g. outbound bandwidth, but that's really hard to estimate. If your traffic is within a VPC it's free.

Your actual numbers will vary but hope that's helpful for thinking through it all.