ML-Based Test Scheduling for Runtime Outlier Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Software testing processes are inefficient due to indiscriminate scheduling of tests, leading to prolonged testing periods and potential failures, as existing methods fail to effectively identify and account for outlying runtime values and prioritize tests based on expected runtimes.
Innovation Solution
A machine learning-based system that applies a trained model to historical runtimes to identify outlying values, determines expected runtimes by excluding these outliers, and schedules tests based on expected runtimes to optimize parallel execution across multiple environments, while monitoring for deviations to prevent failed tests from completing fully.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If tests are scheduled indiscriminately without considering runtime variations, then scheduling simplicity is maintained, but overall testing time is prolonged
Solution Approach 1:
The system performs preliminary analysis of historical runtime data using machine learning models to identify outliers and establish expected runtime ranges before scheduling tests. This advance preparation enables optimized test scheduling that minimizes overall testing time while maintaining reliability.
Solution Approach 2:
The scheduling system dynamically adjusts test execution schedules based on learned runtime patterns and outlier detection. Rather than using static scheduling rules, the system adapts to actual test behavior patterns, optimizing parallel execution and resource allocation to reduce total testing time.
2Measurement precision
If all historical runtime values are used to determine expected runtime, then data completeness is maintained, but accuracy is reduced due to outlying values
Solution Approach 1:
The machine learning model extracts and identifies outlying runtime values from the historical data set, separating them from normal operational variations. By removing these outliers, the system calculates expected runtime based on representative data only, significantly improving accuracy without losing meaningful historical context.
Solution Approach 2:
The system uses feedback from the machine learning model's outlier detection to iteratively refine the expected runtime calculation. The model learns from historical patterns, identifies anomalies, and adjusts the expected runtime determination accordingly, creating a self-improving measurement system.
3Loss of time
If tests with longer runtimes are executed later in the schedule, then resource availability is optimized, but total testing time increases
Solution Approach 1:
The system changes the scheduling parameter from simple chronological or random ordering to runtime-based prioritization. By inverting the traditional approach and scheduling longer tests first rather than last, the system enables better parallel execution planning and reduces total testing time while managing complexity through automated ML-based estimates.
4Reliability
If tests are monitored and terminated early when exceeding expected runtime, then resource waste is prevented, but false positives may occur due to inaccurate runtime estimates
Solution Approach 1:
The system incorporates a safety buffer or threshold in the expected runtime calculation that accounts for natural variability. This cushioning prevents premature termination of valid tests while still enabling early detection of genuinely problematic tests, balancing reliability with resource efficiency.
Data Source
AI summary
A method may include applying to at least a portion of historical runtimes associated with each of a plurality of tests included in a test suite, a machine learning model trained to identify one or more outlying runtime values. The portion of historical runtimes may include an n quantity of the most recent historical runtimes. An expected runtime for each test may be determined based on the portion of historical runtimes excluding the outlying runtime values. A schedule for executing each test in the test suite may be determined based on the expected runtime of each test. The test suite may be executed in accordance with the schedule. Moreover, the executing of the test suite may be monitored based on the expected runtime of each test. Related systems and computer program products are also provided.