Using OpenAI to find out what tests to run

I only experimented in getting newly added tests using git diff and grep, something like git diff -U0 HEAD~1..HEAD '*.test.ts' | ggrep -P '^\+\s*test\(.*' (might not work in your case). Getting modified tests is too complex, especially if you're using fixtures and you modified some fixture. Even for AI you basically would need to give access to whole codebase to deduce what tests were modified.

Thanks for the response. My idea was to make a qualified guess based on the names of the files and functions that were changed. This would help in determining, based on the names of our tests, which tests to run on the PR.

When I talk about changes, I'm not referring to modifications in the test code, but rather changes in the source code of our apps, which the tests are associated with.

sounds interesting. Although have you tried using sharding to make the tests run faster or test out the Playwright service see if it helps you

Usually when creating a testrun for people to run, we have the same problem. It is always a cost/benefit (risk/reward) calculation and if you decide not to run a testcase, and it turns out to have been important, you guessed wrong and you learn from it (it because a part of a required test case).

Asking AI to make the guesses for you is interesting but would be difficult to evaluate the efficacy of its guesses and you would need to have a feedback mechanism to help it learn that it screwed up. That requires humans to help it train its model. And the model that you are creating is highly influenced by the person creating the model (what they think is important, that can differ wildly between people).

Just my 2cents.

We've experimented with the Playwright testing service, but unfortunately, it's proving to be too costly for us at the moment. It's evident that we need to optimize our test speed. Currently, we're using 5 agents in Azure Devops pipelines (with worker set to 1) and running a total of 150 tests. Although all the tests can run in parallel, doing so extends the duration significantly due to the time taken for data setup and other preparatory tasks. Perhaps, to speed things up, we should consider reducing some of the isolation between tests.

@gram7059 why set the worker to 1 in CI, why don't you use parallel test in one CI machine, I think the agent is capable of running multiple parallel tests

I will try it. Last time we got into some issues with it