Test Prioritization via Coverage History Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Dynamic analysis of software code requires significant computing resources and time, making it challenging to run all possible tests during the software development process, necessitating a method to prioritize tests effectively.
Innovation Solution
A method that determines test coverages for multiple tests across different source code versions, stores this information, and prioritizes tests based on weights associated with each test, ensuring that tests covering the most affected code sections are run first, optimizing the detection of defects and vulnerabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all possible tests are run during dynamic analysis, then defect detection coverage is improved, but computing resource consumption and time increase significantly
Solution Approach 1:
The system performs static analysis and determines test coverage information before executing dynamic analysis tests. By pre-calculating which code sections are covered by which tests and identifying affected code sections from version comparisons, the system prepares a prioritization scheme in advance that guides subsequent test execution, avoiding the need to run all tests regardless of their potential value.
Solution Approach 2:
The system changes the parameter of test selection from a static approach (running all tests or a fixed subset) to a dynamic approach where test priority is calculated based on multiple factors including test coverage metrics, code change impact, and historical defect data. This parameter transformation enables adaptive test prioritization that optimizes the balance between coverage and resource consumption.
2Reliability
If comprehensive static analysis is performed to identify all code paths, then detection of rare errors is improved, but analysis time and computational overhead increase
Solution Approach 1:
The system extracts only the necessary test coverage information from static analysis rather than performing exhaustive analysis of all code paths. By focusing on identifying which tests cover which code sections and which sections are affected by changes, the system obtains sufficient information for prioritization without the overhead of comprehensive static analysis of every possible code path.
Solution Approach 2:
The system introduces test coverage information as an intermediary data structure that bridges static analysis results and dynamic test execution. This intermediary layer translates comprehensive static analysis capabilities into practical test prioritization decisions, allowing the system to leverage static analysis insights without directly executing exhaustive code path analysis during the testing phase.
3Productivity
If test prioritization is implemented based on code changes, then testing productivity is improved, but complexity of test management system increases
Solution Approach 1:
The system uses a unified test coverage data structure that serves multiple functions: it tracks which tests cover which code sections, identifies affected code sections from version comparisons, and provides the basis for calculating test priority. This multi-functional approach avoids creating separate complex systems for each function, reducing overall system complexity while maintaining comprehensive test prioritization capabilities.
Data Source
Figure 1
Figure 2~3A
Figure 3B
AI summary
A method is provided to prioritize testing of computer program code comprising: determining first test coverages of items within a first source code version for multiple tests; storing in a non-transitory storage device, a first history that indicates the determined first test coverages of the items within the first source code version; identifying occurrences of the items within a second source code version; determining first weights associated with tests, wherein a respective weight associated with a respective test is indicative of a respective number of respective items within the second source code version that are covered by the respective associated test according to the first history; and prioritizing the multiple respective tests based at least in part upon the determined first weights.