Micro-scheduler for Early Test Breakage Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large-scale software systems with millions or billions of lines of code, identifying the source of test breakages between milestones is time-consuming and resource-intensive, especially when multiple changes occur and interdependencies are high, as existing testing methods require running numerous tests to determine which changes caused the failure.
Innovation Solution
A micro-scheduler system that uses test results from unsubmitted branches to predict test breakages, reschedule tests, and identify which changes have broken tests by evaluating recent test history, using presubmit results to infer post-submit status and determine the range of changes that could be the culprit for a test breakage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional milestone-based testing is used to ensure code reliability, then test coverage is comprehensive, but time consumption and resource usage increase significantly when multiple changes occur between milestones
Solution Approach 1:
The system performs preliminary testing by running tests on presubmit branches before changes are merged to the main codebase. This allows early detection of test breakages caused by individual changes, preventing the accumulation of multiple broken changes that would make breakage source identification difficult and time-consuming at later milestones.
Solution Approach 2:
The system segments the testing process by associating each test with specific change lists and tracking which changes affect which tests. When a test breakage occurs, the system can identify the specific change list responsible by analyzing the segmented test history, rather than having to examine all changes since the last milestone.
2Reliability
If comprehensive automated testing is performed at each milestone to ensure code quality, then test coverage is high, but resource consumption increases significantly
Solution Approach 1:
Instead of running all tests at every milestone, the system performs partial testing by selectively running tests based on change list associations. The system identifies only the tests affected by recent changes and runs those, rather than executing the entire test suite, thus reducing resource consumption while maintaining quality assurance for relevant code paths.
Solution Approach 2:
The system maintains self-service by automatically tracking test history and change list associations in an in-memory grid. This allows the system to autonomously determine which tests need to be run based on presubmit results and change metadata, without requiring manual intervention or comprehensive milestone-based test execution.
3Loss of time
If presubmit testing is used to detect breakages early, then breakage identification time is reduced, but the complexity of tracking and analyzing presubmit results increases
Solution Approach 1:
The system introduces an intermediary in-memory grid that serves as a mediator between presubmit test results and breakage analysis. This grid stores and manages the associations between tests, change lists, and presubmit results, providing a structured interface that simplifies the complexity of tracking and analyzing presubmit data without exposing the underlying complexity to users.
Solution Approach 2:
The system implements feedback mechanisms by continuously monitoring presubmit test results and automatically updating the in-memory grid with new test-metadata associations. This feedback loop allows the system to adaptively track breakages and maintain accurate test history without requiring complex manual analysis, as the system automatically processes and integrates new information.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The technology employs a micro-scheduler arrangement (400) to detect and correct issues relating to software code testing. The micro-scheduler uses test results from unsubmitted branches to predict test breakages, test fixes, and to identify which changes have broken certain tests. This arrangement is high beneficial in large-scale software systems that have millions or billions of lines of code, and can be used to supplement an existing test automation platform. In response to receipt of a new presubmit or postsubmit test result (602), the system evaluates the recent test history in relation to milestone tests in order to determine whether or not to schedule a particular test for one or more change list identifiers (604, 606, 608). Different test conditions, such as currently failing and currently broken, in conjunction with stored presubmit information, help the system determine the range of changes that could be the culprit for a test breakage.