TestCase::onNotSuccessfulTest cannot mark tests as skipped or failed anymore in PHPUnit 10

[stof]

Q	A
PHPUnit version	10.0.7
PHP version	8.1.15
Installation Method	Composer

Summary

In PHPUnit 9 and older, it was possible to override TestCase::onNotSuccessfulTest to change the outcome of a test, as this method was called before determining the test status based on the caught exception (and this method could throw a different one that would be used).
In PHPUnit 10, this method is called after determining the test status, which voids any use case I know for that extension point.

Examples of usage of that method:

• the Mink shared driver testsuite uses it so that an exception marking that a driver does not support that action automatically marks the test as skipped rather than failed or errored: https://github.com/minkphp/driver-testsuite/blob/b6e69945ceb2ec92bf88fa4a04cea92f2e086eab/tests/TestCase.php#L74-L81
• https://www.drupal.org/project/drupal/issues/3271214 show cases where they convert an exception to an AssertionFailedError (which would be better solved by Add TestCase::registerFailureInterface() to register interface that marks exceptions to be treated as failures, not errors #5201 once available)

Current behavior

How to reproduce

1. Create a TestCase with a test throwing an exception
2. Override the onNotSuccessfulTest method in that TestCase to call self::markTestSkipped() when that exception happened
3. Run those tests both with PHPUnit 9 and 10 and compare the difference.

Expected behavior

[sebastianbergmann]

At least for me, there was only ever a single valid use case for this: to have an extension point to "do something" when a test fails. It was never meant to change the outcome of a test. So this is not a bug, it works as intended.

[stof]

What would be the way to do what the driver testsuite does then ?

[stof]

And as this is a behavior change for what is possible with onNotSuccessfulTest, it might make sense to document that in the changelog/upgrade guide for PHPUnit 10

[sebastianbergmann]

Already done in 7142280 :)

[stof]

@sebastianbergmann do you think it is possible for PHPUnit 9.6 to report a deprecation for such usage ?

[sebastianbergmann]

I do not think so.