Test Failure Prediction Using Code Metrics and ML Ranking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing testing methods require substantial computational power and time, making it costly to identify program bugs early in development, and there is a need for more efficient prediction of test failures.
Innovation Solution
A machine learning-based system that evaluates historical code files and test results to predict test failures using hypotheses such as run length, frequency of changes, and user involvement, and provides corrective actions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If thorough tests are run to identify program bugs early in development, then the cost of fixing bugs is reduced, but substantial computational power and time are required
Solution Approach 1:
The system performs preliminary analysis of code metrics and historical test data before executing tests to predict which tests are likely to fail. This preliminary action allows the system to prioritize or skip certain tests, reducing overall testing time while maintaining effective bug detection for high-risk areas.
Solution Approach 2:
Instead of running all tests thoroughly, the system applies partial action by selectively executing only the most critical tests based on failure probability predictions. This partial testing approach reduces computational power and time requirements while still achieving effective bug detection in high-risk code areas.
2Reliability
If thorough tests are run to identify program bugs early in development, then the cost of fixing bugs is reduced, but substantial computational power is required
Solution Approach 1:
The system performs preliminary prediction analysis using machine learning models trained on historical test data and code metrics. This preliminary action identifies high-risk test cases that require full execution, while lower-risk cases can be handled with reduced computational resources or skipped entirely.
Solution Approach 2:
The system applies partial computational effort by focusing full testing resources only on tests predicted to have high failure probability. Tests with low predicted failure rates receive minimal or no computational resources, significantly reducing overall computational power requirements while maintaining effective bug detection.
3Measurement precision
If machine learning engine is trained on historical code files and test results, then test failure prediction accuracy is improved, but training time and computational resources are consumed
Solution Approach 1:
The machine learning training process is segmented into multiple stages: initial training on historical data, validation on separate datasets, and continuous incremental learning. This segmentation allows the system to achieve good prediction accuracy without requiring excessive training time, as the model can be updated incrementally with new data.
Solution Approach 2:
The system applies partial training action by using a subset of the most informative historical features and code metrics for training, rather than processing all available data equally. This selective approach maintains prediction accuracy while reducing training time and computational resource consumption.
Data Source
AI summary
Method for prediction of test failures are disclosed. A method may include retrieving, by a test failure prediction computer program, test results for a test; training, by the test failure prediction computer program, a machine learning engine based on one or more hypotheses; receiving, by the test failure prediction computer program, metrics for a code file; providing, by the test failure prediction computer program, the metrics for the code file to the trained machine learning engine; receiving, by the test failure prediction computer program, an output from the trained machine learning engine that provides a probability of failure; and outputting, by the test failure prediction computer program, the output.