ML Test Scheduling for Hardware-Software Validation Coverage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing product validation methods are inefficient and resource-intensive due to the need to test numerous hardware and software combinations, especially when computing entities have varying configurations, leading to wasted resources on less prevalent and less risky combinations.
Innovation Solution
A test scheduler utilizes machine learning, specifically reinforcement learning, to optimize test combinations by prioritizing those with high prevalence and risk, balancing exploration and exploitation to efficiently allocate resources for feature validation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If numerous performance tests are created and executed to test each product configuration, then testing coverage and reliability are improved, but resource consumption and cost increase significantly
Solution Approach 1:
The patent changes the parameters of test selection by using machine learning models to predict which test combinations are most likely to fail based on historical data. Instead of executing all possible test combinations, the system dynamically adjusts the test portfolio parameters to focus on high-risk configurations, thereby maintaining reliability while reducing resource consumption.
Solution Approach 2:
The patent creates virtual copies of test results through machine learning predictions. Rather than physically executing every test combination on every product configuration, the system uses trained models to generate predicted outcomes for low-risk configurations, effectively copying the value of actual test executions without the associated resource cost.
2Reliability
If all possible hardware and software combinations are tested, then validation completeness is improved, but testing time and productivity are worsened
Solution Approach 1:
The patent applies partial action by selectively testing only the most critical test combinations rather than all possible combinations. The machine learning model identifies a subset of high-value tests that provide the majority of validation coverage, allowing the system to achieve sufficient validation completeness without the excessive time cost of exhaustive testing.
Solution Approach 2:
The patent implements feedback loops where test results from executed combinations are fed back into the machine learning model to refine predictions. This continuous feedback mechanism improves the accuracy of test selection over time, enhancing both validation completeness and testing efficiency by progressively identifying the most informative test combinations.
3Reliability
If test combinations are selected based on uniform distribution, then coverage is improved, but resource efficiency deteriorates due to testing low-prevalence combinations
Solution Approach 1:
The patent applies local quality by making test selection probability depend on the local characteristics of each test combination, specifically its prevalence and risk profile. High-prevalence, high-risk combinations receive higher selection probabilities and more testing resources, while low-prevalence combinations receive fewer resources. This non-uniform distribution optimizes resource efficiency while maintaining adequate coverage.
Data Source
AI summary
Examples of the present disclosure describe systems and methods for selecting test combinations of hardware and software features for product validation. In examples, a test scheduler of a system receives metrics associated with a first batch of test combinations for multiple computing entities. The metrics may include data associated with a fleet prevalence and a fleet risk determined for the multiple computing entities. The scheduler provides one or more of the metrics to a machine learning algorithm. The test scheduler determines a second batch of test combinations associated with a subset of the multiple computing entities based on an output of the machine learning algorithm. The second batch of combinations may include a subset of the first batch of test combinations. The system executes the second batch of test combinations for the subset of the multiple computing entities.