← Back to Discord Forum

different behaviors between debug mode and the test runner

How to handle test pass by debug mode in command line and fail by test runner? Is there any configuration to be done? I already increased the delay and did not work.

This thread is trying to answer question "How to handle a test that passes in debug mode in the command line but fails when run by the test runner?"

10 replies

Debug i would assert will be slower and might be more stable, put in a breakpoint you are implicitly putting in waits... But there should rarely be a difference between passing/failing. Examine the root cause of the failure, as is let's go to Vegas and bet if you or anyone could say without first seeing the code, or one properly doing a root cause of the failure...

Thanks for answer! It's related to the paypal poup in others words, iframe. Its an iframe running inside SDK which open an paypal payment method. It seems that connection is lost when paypal popup is closed. However if I run by debug in command line it passes. I already increased the explicity wait and did not work.

Hmmm, paypal.... iframe, very good chance and you should check if there is a shadow dom. Exactly what one might expect for a payment form...

there is a shadow down, my app runs inside this. paypal came after on payment flow when its back to my app seems that playwright lost the connection to the browser

weird thing that it runs by --debug tag

is the shadow dom open or closed, if closed not sure how you'll get past it...

opened for instance, maybe its reloaded as soon as paypal popup is closed

Hmm then it isn't a difference between debug? Vs the browser dying...

that's what I want to identify.. sometimes show the browser closed error or timeout

I am guessing that plawirhg lost some connection in runner mode, since it runs in debug mode


Debugging Test Failures in @playwright/test

When your tests pass in debug mode but fail when run by a test runner, it can be due to differences in timing, environment setup, and configuration.

Timing Issues

Debug mode allows you to step through your code and pause at breakpoints. This can affect the timing of actions or assertions in your tests. When tests are run by a test runner without debugging, they execute at their normal pace. If you suspect timing issues, consider adding explicit delays or waits in your tests.

import { test } from '@playwright/test';

test('example test', async ({ page }) => {
  await page.goto('https://ray.run/');
  await page.waitForTimeout(2000); // explicit delay
  // rest of your test

Environment Setup

Ensure that your environment setup is consistent when running tests in debug mode and with a test runner. This includes configurations or dependencies that might be set up differently.

Configuration Settings

Review your configuration files and ensure that all relevant settings are properly configured for both scenarios.

import { test } from '@playwright/test';

test.setTimeout(20000); // set timeout

Debugging Steps

  1. Review your codebase for differences in handling actions or assertions during debugging versus regular execution.
  2. Analyze logs and error messages that occur only during non-debug runs.
  3. Enable detailed logging during test runs with the test runner.
  4. Try to replicate the failure scenario on your local machine using the same test runner and configuration.
  5. Discuss the issue with your team members.

Remember, debugging requires a systematic approach. Gather as much information as possible, analyze it thoroughly, and make targeted adjustments based on your findings. For more tips, check out Mastering the Art of Detecting and Handling Flaky Tests in Playwright.

Related Discord Threads

AboutQuestionsDiscord ForumBrowser ExtensionTagsQA Jobs

Rayrun is a community for QA engineers. I am constantly looking for new ways to add value to people learning Playwright and other browser automation frameworks. If you have feedback, email luc@ray.run.