Dynamic Test Case Scheduling for Software Regression Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing software testing methodologies are inefficient, particularly in large and complex applications, as they require executing numerous test cases across various devices, leading to resource bottlenecks and potential deployment of bad code due to untimely testing, especially in continuous development environments where multiple developers check in code simultaneously.
Innovation Solution
A dynamic scheduling system that prioritizes test cases based on when they were last executed, using factors such as priority multiplication, maximum time since last execution, and maximum number of commits since last execution, to optimize resource usage and minimize regression detection time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all test cases are executed on every commit, then reliability is improved, but productivity deteriorates due to excessive execution time and resource consumption
Solution Approach 1:
The patent extracts only the necessary subset of test cases that are relevant to the current commit changes, rather than executing all test cases. This is achieved by analyzing commit metadata and test case dependencies to identify which specific test cases need to be run, thereby reducing execution time while maintaining quality assurance.
Solution Approach 2:
The patent implements dynamic scheduling of test cases based on real-time factors such as commit type, code changes, and test case priorities. The test execution plan is dynamically adjusted for each commit rather than following a static all-test-cases approach, optimizing the balance between reliability and productivity.
2Reliability
If test cases are executed frequently, then regression detection is improved, but loss of time increases due to repeated execution of unchanged test cases
Solution Approach 1:
The patent extracts and executes only those test cases that are relevant to detecting regressions in the current commit, excluding test cases that are not affected by the changes. This selective approach maintains regression detection capability while eliminating wasted execution time on unrelated test cases.
Solution Approach 2:
The patent changes the execution parameters of test cases based on commit characteristics. Test cases are dynamically selected and scheduled based on parameters such as the type of code change, files modified, and test case dependencies, optimizing the frequency and timing of executions to minimize time loss while maintaining detection effectiveness.
3Productivity
If multiple developers check in code simultaneously, then productivity is improved through parallel development, but device availability deteriorates due to resource bottlenecks
Solution Approach 1:
The patent segments the test case execution workload across multiple devices and developers simultaneously. Each commit is independently evaluated and scheduled on appropriate available devices, allowing parallel processing of multiple test suites without resource bottlenecks. The system divides the overall testing task into smaller, manageable segments that can be executed concurrently.
Solution Approach 2:
The patent implements dynamic resource allocation that adapts to the current state of device availability and commit queue. When multiple developers are checking in code simultaneously, the system dynamically assigns test cases to available devices based on real-time capacity, preventing resource bottlenecks while maintaining high development velocity.
4Ease of operation
If test cases are scheduled based on fixed intervals, then ease of operation is improved, but adaptability deteriorates in response to changing test-commit load and device capacity
Solution Approach 1:
The patent replaces fixed-interval scheduling with a dynamic scheduling system that automatically adapts to changing conditions. The system monitors device capacity, commit frequency, and test case priorities in real-time, adjusting the execution schedule dynamically. This maintains ease of operation through automated decision-making while achieving high adaptability to dynamic environments.
Solution Approach 2:
The patent incorporates feedback mechanisms that monitor the state of the testing system and adjust scheduling decisions accordingly. The system receives feedback on device availability, commit patterns, and test execution results, using this information to dynamically optimize the test case schedule. This feedback loop enables the system to adapt to changing conditions automatically while maintaining operational simplicity.
Data Source
AI summary
Dynamically scheduled testing of a software application on one or more devices are disclosed. The system can schedule and execute of one or more test cases associated with a software application. When a device becomes available for testing, the system can determine the list of commits that have not yet been tested. If there are outstanding untested commits, the system can determine the list of test cases in a test plan that are to be scheduled and executed. For each test in the list of test cases, the system can compute an urgency value. Urgency value can be a function of the untested commits. For example, urgency value can be the number of untested commits associated with each test case. The system can then sort the test cases based on the urgency value and schedule the most “urgent” test case for execution.