Incident Tracker for Automated Regression Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing regression testing techniques are time-consuming and inefficient in identifying regression incidents, especially when multiple revisions are made to software code in a short duration, failing to effectively detect and isolate the revisions causing these incidents.
Innovation Solution
An incident tracker system and method that evaluates source code revisions using metric values and regression detection techniques like change point analysis and simulation analysis to automatically detect and identify the revisions causing regression incidents, and subsequently identifies revisions that fix these issues, enabling efficient reporting and workflow management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional regression testing techniques are used to evaluate software changes, then comprehensive bug detection is achieved, but the testing process becomes time-consuming and development cycles are prolonged
Solution Approach 1:
The patent segments the regression testing process by dividing test cases into different priority levels and categories. Instead of executing all test cases uniformly, the system segments them based on their relevance to the specific code changes, allowing critical tests to be executed while less relevant tests are skipped or deferred, thus reducing overall testing time while maintaining bug detection effectiveness.
Solution Approach 2:
The patent implements dynamic test case selection and execution prioritization based on the nature and impact of code changes. The system dynamically adjusts which test cases to execute and in what order, rather than following a static predetermined sequence. This dynamic approach allows the testing process to adapt to specific change scenarios, optimizing both time efficiency and detection reliability.
2Reliability
If comprehensive regression testing is performed on all code revisions, then all regression incidents are detected, but the complexity of managing and analyzing test results increases significantly
Solution Approach 1:
The patent extracts and focuses on the most relevant test results by filtering out noise and irrelevant data. The system identifies and prioritizes test cases that are most likely to detect regression incidents based on the code changes, extracting only the critical information needed for effective analysis. This reduces the volume of test results that need to be managed and analyzed, simplifying the overall process while maintaining detection effectiveness.
Solution Approach 2:
The patent implements feedback mechanisms that automatically analyze test results and provide actionable insights. The system uses feedback from previous test executions and code change patterns to automatically adjust test selection, prioritize results, and even identify potential regressions without requiring manual analysis of all test outcomes. This automated feedback loop reduces the complexity of test result management while maintaining high detection reliability.
3Productivity
If multiple test cases are executed in parallel to reduce testing time, then productivity increases, but the difficulty of identifying which specific revision caused a regression incident increases
Solution Approach 1:
The patent introduces an intermediary layer that coordinates and tracks parallel test executions. This intermediary system maintains the relationship between code revisions and test results, even when tests are executed in parallel. It uses version control integration and test result tagging to ensure that each test result can be traced back to the specific code revision being tested, thereby maintaining source identification capability while enabling parallel execution for improved productivity.
Solution Approach 2:
The patent replaces manual tracking and analysis mechanisms with automated computational systems. Instead of manually tracking which revision caused which regression across parallel test executions, the system uses automated algorithms to analyze test results, correlate them with code changes, and identify the offending revision. This substitution of mechanical/manual processes with automated computational analysis maintains source identification accuracy while enabling high-throughput parallel testing.
Data Source
AI summary
A system and method that automatically detects that a regression incident has occurred, and identifies the particular revision(s) of source code that introduced the regression incident is disclosed herein. The system and method execute one or more simulations of each revision in a window of revisions of the source code to compute values for one or more metrics. The metric values are then used to compute regression confidence values as well as a regression confidence interval—to then detect a regression incident and identify the revision that caused the regression incident. The system and method can also identify subsequent revision(s) that sufficiently fix regression incident(s) caused by a revision such that a software developer can be informed of these fixing revisions.