[Feature]: Add failureThreshold config

[brunojbarros]

🚀 Feature Request

Add the ability to configure tests to stop running if a failure threshold is reached, those that do not run should be skipped.

Tests marked with "skip" should not be considered for the percentage calculation.

Things to consider:

1. It doesn't make sense to have maxFailures and failureThreshold (possible new setting) in effect at the same time, but if set it could pause testing once the first setting is reached.
2. Tests marked with "skip" should not be considered for the percentage calculation.

What is the best way to act in case "failureThreshold" is set in defineConfig and also in defineConfig.projects[]?

For example, given 100 tests, of which 25 belong to project "chromium", would the total number of tests on which defineConfig.failureThreshold will perform the calculation be 100 or 75?

Perhaps you could override the value of defineConfig.failureThreshold and only use defineConfig.projects[] when running a specific project... (thinking out loud).

Example

export default defineConfig({
  // Once 10% failure is reached, subsequent tests should be skipped
  failureThreshold: 10,
  // ...
  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
      // It will use failureThreshold: 10
    },
    {
      name: 'webkit',
      use: { ...devices['Desktop Safari'] },
      failureThreshold: 25,
    },
  ],
});

Motivation

Currently is possible to add a maxFailures to stop Playwright Test will stop after reaching this number of failed tests and skip any tests that were not executed yet.

It is not the same maxFailures = 5 when you have only 50 tests (failure threshold = 10%) or 5000 tests (failure threshold = 0,1%).

[pavelfeldman]

maxFailures: 50 is not about allowing 50 tests to fail, it is rather about quitting early when all tests fail due to a severe underlying condition and saving on CI cost. Is the spirit of the suggested failureThreshold the same? I.e. are you suggesting to have a way to specify maxFailures in %? Not sure it makes sense.