Intelligent Test Case Prioritization via Failure Propensity Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Software testing and certification processes are time-consuming, leading to delays in making software applications available to users, and often require prioritization of test cases to address potential failures effectively.
Innovation Solution
A deep learning model is trained to predict a failure propensity score for each test case, allowing for dynamic prioritization and scheduling of test cases based on their likelihood of failure, using execution and change history information to generate model inputs and determine testing schedules.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If comprehensive software testing is performed on all test cases, then software reliability is improved, but testing time and productivity are worsened
Solution Approach 1:
The system performs preliminary analysis of code changes and executes high-priority test cases first based on predicted failure propensity. By identifying and executing critical tests before lower-priority ones, the system ensures that most reliability concerns are addressed early, allowing optional skipping of lower-priority tests to improve productivity without significantly compromising overall software reliability.
2Reliability
If all test cases are executed, then software reliability is improved, but time consumption is worsened
Solution Approach 1:
The system performs preliminary prediction of test case failure propensity using machine learning models that analyze code changes, test case characteristics, and historical execution data. This preliminary assessment enables prioritization before actual test execution, allowing the system to focus time on high-risk tests while maintaining reliability through targeted testing of critical paths.
Solution Approach 2:
The test case prioritization is dynamic and adapts based on current code changes, test results, and historical data. The system continuously updates failure propensity predictions as new information becomes available during the testing process, allowing flexible adjustment of test execution order to optimize both reliability and time consumption based on actual testing progress.
3Productivity
If test cases are prioritized based on failure propensity, then testing efficiency is improved, but computational resources are worsened due to model training
Solution Approach 1:
The machine learning model for predicting test case failure propensity is trained in advance using historical test execution data and code change information. This preliminary training phase creates a reusable model that can quickly prioritize test cases without requiring intensive computational resources during actual test execution, thus improving testing efficiency while managing resource consumption through batch processing during the training phase.
Data Source
AI summary
Techniques for intelligent application source code testing prioritization are disclosed. In one embodiment, a method is disclosed comprising obtaining execution history information associated with a number of test cases and source code change history information, generating a plurality of encodings for a number of test cases, each test case encoding comprising a merger of a test case description and source code change history information corresponding to a test case, determining a failure propensity score for each test case using a trained failure prediction model and the test case's encoding, determining a prioritization of the test cases in accordance with the failure propensity scores, such that a test case having a greater failure propensity score indicating a greater propensity to fail having a higher priority than another test case with a lower propensity to fail, and generating a testing schedule for the plurality of test cases in accordance with the test case prioritization.