Rayrun
โ† Back to Discord Forum

[Resolved]: Failed test logged as a passed

Hi,

I have the following test:

import { chromium, test, expect, Browser, Page } from '@playwright/test';
import * as TE from 'fp-ts/TaskEither';
import * as F from 'fp-ts/function';

const launchBrowser = (): TE.TaskEither<Error, Browser> => 
  TE.tryCatch<Error, Browser>(
    async () => chromium.launch(),
    (error: any) => new Error(error)
  );

const newPage = (browser: Browser): TE.TaskEither<Error, Page> => 
  TE.tryCatch<Error, Page>(
    async () => browser.newPage(),
    (error: any) => new Error(error)
  );

const goto = (page: Page): TE.TaskEither<Error, Page> =>
  TE.tryCatch<Error, Page>(
    async () => {
      await page.goto('https://playwright.dev/');
      return page;
    },
    (error: any) => new Error(error)
  );

test('My first test with fp approach', async () => {
  await F.pipe(
    launchBrowser(),
    TE.chain((browser) => newPage(browser)),
    TE.chain((page) => goto(page)),
    TE.map(async (page) => expect(page).toHaveTitle(/sss/))
  )();
})

When I run this test with the npm or VS Code extension, I got the next result:

In the terminal:

Error: expect(locator).toHaveTitle(expected)

Locator: locator(':root')
Expected pattern: /sss/
Received string:  ""
Call log:
  - expect.toHaveTitle with timeout 5000ms
  - waiting for locator(':root')


Failed worker ran 1 test:
[chromium] โ€บ tests\example.spec.ts:26:5 โ€บ My first test with fp approach

  34 |

    at .\my-playwright-project-fp\tests\example.spec.ts:31:41
    at .\my-playwright-project-fp\node_modules\fp-ts\lib\Either.js:349:62
    at pipe (.\my-playwright-project-fp\node_modules\fp-ts\lib\function.js:299:20)
    at Object._map (.\my-playwright-project-fp\node_modules\fp-ts\lib\Either.js:63:58)
    at .\my-playwright-project-fp\node_modules\fp-ts\lib\Functor.js:19:92
    at .\my-playwright-project-fp\tests\example.spec.ts:27:3

Waiting for the debugger to disconnect...

  1 passed (1.7s)
  1 error was not a part of any test, see above for details

To open last HTML report run:

  npx playwright show-report

In the html report, it looks like on the 1st snapshot In the VS Code extension it looks like on the 2nd snapshot

Does anybody have idea what it's such weird behavior? ๐Ÿ™‚

image.png
image.png

This thread is trying to answer question "Why is a failed test being logged as passed when run with npm or VS Code extension?"

8 replies

this code is working now:

import { chromium, test, expect, Browser, Page } from '@playwright/test';
import * as TE from 'fp-ts/TaskEither';
import * as F from 'fp-ts/function';

const launchBrowser = (): TE.TaskEither<Error, Browser> => 
  TE.tryCatch<Error, Browser>(
    async () => chromium.launch(),
    (error: any) => new Error(error)
  );

const newPage = (browser: Browser): TE.TaskEither<Error, Page> => 
  TE.tryCatch<Error, Page>(
    async () => browser.newPage(),
    (error: any) => new Error(error)
  );

const goto = (page: Page): TE.TaskEither<Error, Page> =>
  TE.tryCatch<Error, Page>(
    async () => {
      await page.goto('https://playwright.dev/');
      return page;
    },
    (error: any) => new Error(error)
  );

const verifyTitle = (page: Page): TE.TaskEither<Error, string> =>
  TE.tryCatch<Error, string>(
    async () => page.title(),
    (error: any) => new Error(error)
  );

test('My first test with fp approach', async () => {
  await F.pipe(
    launchBrowser(),
    TE.chain((browser) => newPage(browser)),
    TE.chain((page) => goto(page)),
    TE.chain((page) => verifyTitle(page)),
    TE.map((title) => expect(title).toContain('asdasdsad'))
  )();
});
1) [chromium] โ€บ example.spec.ts:34:7 โ€บ First โ€บ My first test with fp approach โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€

    Error: expect(received).toContain(expected) // indexOf

    Expected substring: "asdasdsad"
    Received string:    "Fast and reliable end-to-end testing for modern web apps | Playwright"

      38 |       TE.chain((page) => goto(page)),
      39 |       TE.chain((page) => verifyTitle(page)),
    > 40 |       TE.map((title) => expect(title).toContain('asdasdsad'))
         |                                       ^
      41 |     )();
      42 |   });
      43 | });
refactoreric

Hmm that code is way over my head.

But just want to warn you that if you don't use/rely on the built-in fixtures to create and tear down the browser, context and page, some things like configuring trace files, recording and screenshots in the playwright.config may not always work as expected.

(And maybe the vscode extension might have trouble with this too)

And be aware that primitive matchers from Jest don't have the auto-retrying feature which is so great in Playwright. For example instead of expect(title).toContain, you would ideally write await expect(page).toHaveTitle

These are the auto-retrying assertions which are best-practice to use, to make tests less flaky: https://playwright.dev/docs/test-assertions#auto-retrying-assertions

@refactoreric thank you,

but I know that, but it does not work as you can see from the first part of code. If I return the title with await page.title(), then chaeck it as expect(title).toContain('Playwright') it works. I have no idea why the await expect(page).toHaveTitle(/Playwright/) does not work in the map(async (page) => expect(page).toHaveTitle(/Playwright/)) code

You are not awaiting expect(page).toHaveTitle(/Playwright/)). I suspect Playwright begins worker teardown, reporter exiting, etc and marks the test as passed because expect hasn't resolved yet at that moment.

@skorp32: why not await? It's awaited

why not await? It's awaited

image.png

Ehh, I see no await before expect I complain ๐Ÿ˜… guess this code is over my head too

@skorp32 @refactoreric I found the reason why it's failed. Cuz I return the TE.TaskEither<Error, void>

const verifyTitle = (page: Page): TE.TaskEither<Error, void> =>
  TE.tryCatch<Error, void>(
    async () => await expect(page).toHaveTitle(/ssss/),
    (error: any) => new Error(error)
  );

that means that this error will newer be thrown. What I need to change just this lines

const verifyTitle = (page: Page): TE.TaskEither<void, void> =>
  TE.tryCatch<void, void>(
    async () => await expect(page).toHaveTitle(/ssss/),
    (error: any) => {
  throw new Error(error as string);
}
  );

[Resolved]: Failed test logged as a passed

Related Discord Threads

TwitterGitHubLinkedIn
AboutQuestionsDiscord ForumBrowser ExtensionTagsQA Jobs

Rayrun is a community for QA engineers. I am constantly looking for new ways to add value to people learning Playwright and other browser automation frameworks. If you have feedback, email luc@ray.run.