Failing Test Generation from Proof Counterexamples
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge lies in bridging the gap between program proofs and the identification of specific test cases that cause a program to fail, as program testing is easier to conduct but does not guarantee correctness, while program proving guarantees correctness but is difficult to obtain successfully.
Innovation Solution
A method is introduced that involves verifying computer-executable instructions using a program-proving module with predefined verification conditions. If the proof is unsuccessful, a counterexample is generated, and a failing test is created by a test-case generator based on this counterexample to help identify and rectify bugs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If program testing is used to verify software correctness, then it is easier to conduct and provides concrete failing test cases, but it does not guarantee correctness and may miss edge cases
Solution Approach 1:
The patent merges program testing and program proving into a unified verification framework. The testing module executes the program with generated test cases, while the proving module formally verifies correctness. Both modules work together to provide both concrete failing test cases and correctness guarantees, resolving the contradiction between ease of operation and reliability.
Solution Approach 2:
The patent implements feedback mechanisms where the testing module provides concrete counterexamples to the proving module, and the proving module provides formal verification results back to the testing module. This feedback loop allows the system to leverage the strengths of both testing (concrete failing cases) and proving (correctness guarantees) to resolve the contradiction.
2Reliability
If program proving is used to verify software correctness, then it guarantees correctness through mathematical proof, but it is difficult to obtain successfully and requires sophisticated tools
Solution Approach 1:
The patent introduces an intermediary testing module that bridges the proving module and the programmer. The testing module generates concrete test cases from the program and verification conditions, executes them, and provides counterexamples when proofs fail. This intermediary simplifies the proving process by providing concrete guidance and reducing the complexity burden on the programmer.
Solution Approach 2:
The patent creates a simplified copy of the program in the form of test cases that can be easily executed and analyzed. Instead of working directly with the complex original program during proof verification, the system uses simplified test case representations that capture essential behaviors, making the proving process more manageable.
3Reliability
If formal verification conditions are generated and proved, then correctness can be mathematically established, but the number of verification conditions becomes astronomically large and impractical
Solution Approach 1:
The patent extracts only the essential verification conditions from the vast set of possible conditions. By using the testing module to generate concrete test cases and identify critical verification conditions, the system filters out redundant conditions and focuses proof efforts on the most important ones, reducing the quantity from astronomical to manageable levels.
Solution Approach 2:
The patent applies partial verification by focusing on specific verification conditions derived from actual test cases rather than attempting to verify all possible conditions. This partial action approach verifies the most critical aspects of correctness while avoiding the impractical task of exhaustive verification of all conditions.
Data Source
AI summary
A system verifies, utilizing at least a hardware processor, at least one computer-executable instruction of a program using a verification condition. In response to determining that the computer-executable instruction fails the verification condition, the system generates, utilizing at least the hardware processor, a counterexample that comprises an execution trace indicative of a proof failure corresponding to the verification condition. The system generates, utilizing at least the hardware processor, a failing test for debugging the program based on the counterexample.