This thread is trying to answer question "Is it possible to use Playwright for performance testing and what are the alternatives and costs?"
I think you can look into k6 or artillery. I have used k6 and it is a great tool for performance testing. What is more have a look at it. As this mighty https://k6.io/blog/get-started-with-k6-browser/#writing-the-test
This can be used as an inspiration https://www.youtube.com/watch?v=IrK-XDH72bw Didn't have time to try myself or play with it, but... looks like it's going to work
yes, this is great video, he made repo for JS https://github.com/unlikelyzero/js-perf-toolkit
👋 member of team Artillery here. Playwright + Artillery scale pretty darn well for load testing, certainly better than anything else out there as we let you use AWS directly! What sort of web performance tests are you looking to set up @doublez? something like tracking Core Web Vitals on certain pages over time?
I have a following scenario. Do you guys think it make sense to use k6 or artillery for it, or what approach do you recommend? (We are using Playwright for frontend E2E tests so it would make sense for me to reuse them as much as possible)
The problem: Some clients are complaining that our app takes to much time to load. Devs want to do some upgrade. Unfortunatelly as a QA I don't have any metrics how much time it takes for the app to load and users can start do their thing. And I want to get something in my hand to evaluate the upgrade. I don't think it is necesary to do load testing in a form of opening the app thousands times in parallel. I would be happy to do it hundred times in serial with my local machine. But I need to get some meaningful results and metrics. I don't think it make sense just to record test result run time, because it can be pretty random as it would also account for browser start, test setup and teardown...
@darill This can be done with Artillery. We have an example here that shows how to track metrics that are specific to the app you're testing (so you would only measure in-app actions, and not browser startup time etc): https://github.com/artilleryio/artillery/tree/main/examples/browser-load-testing-playwright#example-3-tracking-custom-metrics-for-part-of-the-flow
You can also configure Artillery to run one instance of Playwright at a time, and create a new one as soon as the previous one finishes with a
phases configuration that looks like this:
hi @hveldstra , I am considering using artillery on a new project that has come up. I need to saturate an ec2 instance to know how many users we can support implementing a series of steps to get an initial baseline. https://github.com/artilleryio/artillery/tree/main/examples/browser-load-testing-playwright what is the median cost to run this on fargate in your production workloads?
I found https://www.artillery.io/blog/this-load-test-cost-us-how-much. However this is aws lamda and docs does not list playwright in supported engines. If we ran the same 15 minutes on fargate what would be the cost?
Fargate is very cheap too @leblancmeneses. Artillery uses 4 vCPU/8GB RAM for each worker by default. Taking
us-east-1 pricing it'll cost:
0.04048 * 4 + 0.004445 * 8 = $0.19748 / hour per worker
How much load each worker can generate depends on your test scripts. For simplicity's sake let's say we can run one Chrome instance for each vCPU. That would mean that each Artillery worker could run up to 4 concurrent browsers at a time. With 100 workers you could run up to 400 concurrent virtual users. Let's say each VU session lasts 3 minutes. That's means that 100 Artillery workers, running 4 VUs concurrently each for an hour would give you: 60/3* 4 * 100 = 8k total virtual users in one hour, which would cost $19.7 in Fargate fees.
You can also customize the number of vCPUs per worker on Fargate up to 16, which would let you run more browsers concurrently (at higher cost).
There's more to total cost than just direct Fargate costs, e.g. outbound bandwidth, but that's really hard to estimate. If your traffic is within a VPC it's free.
Your actual numbers will vary but hope that's helpful for thinking through it all.
Rayrun is a community for QA engineers. I am constantly looking for new ways to add value to people learning Playwright and other browser automation frameworks. If you have feedback, email email@example.com.