How to solve FLAKY tests?
Hello, guys! I am writing e2e tests for magento based websites. I have about 37 tests and 2 or 3 are usual fails. But they are not the same all the time. Most of reason System.TimeoutException : Timeout 30000ms exceeded. What is the best practice in solving these issue? WE want to add script with retry on CI for such kind of tests. Please, advice if this is on solution? Thanks in advance!
This thread is trying to answer question "What is the best practice in solving flaky tests in e2e tests for Magento based websites?"
Hi, the first thing I recommend is enabling recording and publishing of trace files for failed tests.
Failed tests in CI are much easier to troubleshoot when having trace files. In NodeJS (@playwright/test) I set the traces to 'retain-on-failure'.
https://playwright.dev/dotnet/docs/trace-viewer-intro
With trace files you know what state the UI was in when it was waiting for some element, including a DOM snapshot that you can inspect, you know what JavaScript errors occured, which network requests took place, what responses they got, etc. It removes a lot of guessing from the troubleshooting process.
What to so if test fails in different place from time to time? And it seems that element wasn't load or so. I have added some checks to weak places, but i can't to use it everywhereš¢
Yep, it might be that the test machine is too busy and some pages under test are getting so slow they hit the 30s time out.
How many workers do you use, and how many cores does your test machine have? Or are you aware of some other heavy process going on during your tests?
Also to alleviate these seemingly random failures, you could add more expectations, for intermediate states of the elements and network
Rayrun is a community for QA engineers. I am constantly looking for new ways to add value to people learning Playwright and other browser automation frameworks. If you have feedback, email [email protected].