Cake *.WithRetry test targets exit 0 when a test fails on both the run and the retry #1369

Summary

The Cake *.WithRetry test targets exit 0 even when a test fails on both its first run and its retry. CI shows the job as green, so a real, reproducible test failure can go unnoticed. All the main test workflows use these targets:

The UTS workflow (run-tests-uts.yml) is not affected, because its targets have no retry.

Where it happens

• The task: in cake-build/tasks/test.cake L108–L130, the first run's failure is caught and turned into a warning, and the retry is called after it:

try
{
    testExecutionHelper.RunDotNetTests(project, settings);
}
catch
{
    Warning("Some tests failed. Retrying failed tests...");
}

testExecutionHelper.RetryFailedDotNetTests(project, resultsPath, testRetryHelper, framework, configuration);

• The retry: in cake-build/helpers/test-execution.cake L233–L276 (RetryFailedDotNetTests), each retried test's failure is also caught and only logged:

catch
{
    _context.Warning($"Test {test} failed on retry");
}

• The .NET Framework path: RetryFailedXUnitTests (L193–L226) and the _NetFramework_*_WithRetry tasks follow the same pattern.

Nothing is rethrown, so the Cake task always succeeds.

Observed

During local validation for #1364, ./build.sh --target=Test.NetStandard.Unit.WithRetry --framework=net6.0 exited 0. In that same run, ConnectionFailureSpecs.WhenInDisconnectedState_ReconnectUsingIncrementalBackoffTimeout failed on both the first run and the retry.

Expected

A test that fails on the first run is retried once. If it fails again, the target fails with a non-zero exit code and the CI job goes red.

Suggested fix

• Make the retry fail the build: have RetryFailedDotNetTests / RetryFailedXUnitTests collect the tests that still fail and throw when that list is non-empty, e.g. throw new Exception($"{n} test(s) failed after retry: …").
• Handle a first run that fails with nothing to retry: for example a crash, or no .trx file written. Check that this case is not also treated as success (FindFailed*Tests returning an empty list).
• Check how much this has hidden: after the change, re-run CI on main to see whether any tests have been failing silently.

┆Issue is synchronized with this Jira Bug by Unito