ML-Based Test Selection for Faster Automated Code Validation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As code bases grow, the number of automated tests increases, leading to inefficiencies due to many tests not providing meaningful information and consuming unnecessary computing resources, as they often test unrelated functionality unaffected by code changes.
Innovation Solution
A machine learning model is trained to identify the likelihood of test failure based on code change characteristics and developer feedback, selecting a subset of tests to run, reducing the number of tests by up to 90% while ensuring meaningful information is obtained.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all automated tests are run against code changes in a large code base, then test coverage is improved, but testing time and computing resources are wasted on tests that will not be affected by the code changes
Solution Approach 1:
The patent segments the complete test suite into relevant and irrelevant tests based on the code change. The test selection system divides the testing process by identifying only those tests that are affected by the specific code change, rather than executing the entire test suite. This segmentation allows the system to maintain comprehensive coverage of affected functionality while eliminating waste on unrelated tests.
Solution Approach 2:
The patent applies partial action by running only the necessary subset of tests rather than the complete test suite. The test selection system determines that running all tests is excessive for most code changes, and instead executes only the partial set of tests that are actually relevant to the code change, thereby reducing testing time while maintaining adequate coverage.
2Reliability
If all automated tests are run against code changes, then test completeness is improved, but computing resources are wasted on tests unrelated to the code changes
Solution Approach 1:
The test selection system segments the test suite by analyzing the code change and identifying which tests are relevant. This segmentation separates affected tests from unaffected tests, allowing the system to execute only the necessary portion while maintaining completeness for the affected code paths.
Solution Approach 2:
The patent extracts the relevant subset of tests from the complete test suite based on the code change characteristics. By taking out only the necessary tests for execution while excluding unrelated tests, the system maintains test completeness for affected functionality while conserving computing resources.
3Manufacturing precision
If a large number of tests are run on code changes, then test thoroughness is improved, but testing efficiency deteriorates due to running tests on unrelated functionality
Solution Approach 1:
The test selection system dynamically adjusts the test suite based on the specific code change. Rather than using a static complete test suite for all changes, the system adapts by selecting only the relevant tests for each code change, thereby maintaining thoroughness for affected code while improving overall testing efficiency.
Solution Approach 2:
The patent changes the parameter of test selection from fixed (all tests) to variable (selected tests based on code change characteristics). By changing how tests are selected based on parameters such as code change location, type, and impact, the system maintains thoroughness where needed while improving efficiency overall.
Data Source
AI summary
A machine learning model is trained from characteristics of code changes and characteristics of tests to generate an output indicative of a likely test result of running a corresponding test on a code change. One or more machine learning models may be trained for a specific code repository and based on developer feedback. When a code change is generated by a developer to code in a code repository, a machine learning model is selected based on the repository and characteristics or features of the code change are extracted and input to the machine learning model. The machine learning model generates a model output indicative of the likely test results of running each of a plurality of different tests on the code change. The model output indicates how likely it is that each of the plurality of different tests will fail. Based on the model output, a test selection system selects a subset of the plurality of different tests that should be run against the code changes.