Backward Scoring Ranks Test Suites by Real Incidents Caught

When a test lacks catch history or a preventive story, it may be flagged for potential retirement. The conversation that follows this flagging is more important than the flag itself, guiding the decision-making process.

The workflow assigns an owner to the flagged test, initiates a review process, and leads to a final outcome of either retirement or relabeling, ensuring that tests are not discarded prematurely based solely on an 'unproven' verdict.

Traditional coverage thresholds often focus on quantitative metrics like code and branch lines, which can encourage writing more tests without necessarily improving their effectiveness. Backward scoring directly addresses the problem of incident prevention by shifting the focus to the outcome of proven catches.

This method encourages developing tests that actively prevent incidents rather than merely increasing coverage percentages, ensuring that resources are allocated to truly effective tests.

Five essential points should be considered for improving test suites, including the need to separate discussions on coverage percentage from incident prevention. Coverage is closely related to meeting requirements, while incident prevention is tied to defect density and the overall effectiveness of tests.

Backward scoring helps validate candidate tests by replaying real incidents against them, providing a clear measure of their actual utility in preventing future issues.

Approximately one-fifth of 4,200 tests reviewed showed no unique historical catch evidence, resulting in an 'unproven' verdict. This outcome can mean a test is either dead weight or a quietly effective preventive measure performing its job by stopping bugs before they are logged.

It is crucial not to delete tests based solely on an 'unproven' verdict, as these tests require a thorough owner review to determine their actual value and potential for preventing future incidents.

A 'proven' test is one that has failed on at least one incident that no earlier test had previously caught, demonstrating its unique effectiveness.

A 'duplicate' test refers to real signals that follow an upstream test which has already flagged the same issue, indicating redundancy. An 'unproven' test, by contrast, is one with zero historical catches, necessitating further investigation into its utility.

The backward scoring method begins by recording an incident in the database to establish a baseline. The system is then reverted to its state immediately before the first symptom of the incident appeared.

Every candidate test is replayed against this pre-incident state to observe its behavior. The failures are subsequently tallied into a verdict, providing clear evidence of which tests would have successfully caught the incident.

One limitation is the preventive blind spot: tests that prevent a bug from being logged can still be proven effective, even if they don't appear in incident logs. This indicates that their true value might not always be immediately visible through direct incident correlation.

Another limitation is the method's dependency on incident logs. An incident that is never logged cannot, by definition, be caught by this system, potentially leading to an incomplete picture of test effectiveness. This means that tests that prevent issues before they become formal incidents may appear less valuable.

Logs themselves can also make valid tests appear unproven if incidents are not meticulously recorded or if the context of the incident is lost. The accuracy of the backward scoring method heavily relies on the quality and completeness of incident logging.

Therefore, while this preventive mechanism helps ensure proven results, it is essential to recognize the inherent biases and data dependencies that can influence the classification of test verdicts.