Heuristics for Automated Culprit Finding in Codebases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large engineering projects with rapidly changing codebases, identifying the specific code changes that cause test failures is difficult and time-consuming due to the lack of information from automatic test platforms about which changes led to errors, requiring manual investigation by programmers.
Innovation Solution
A system and method that utilize a heuristic approach to isolate code changes causing failures by calculating the impact of each change on test targets, generating a dependency tree, and sorting changes based on impact, with optional use of compile or execution logs to determine failure causes and identify root causes, implemented in a backend, frontend, and datastore architecture.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual investigation is used to identify culprit code changes, then accuracy in finding the root cause can be maintained, but time consumption increases significantly
Solution Approach 1:
The patent segments the large set of code changes into smaller subsets using multiple sorting heuristics. Each heuristic processes a portion of the changes and produces a sorted list of suspected culprits. This segmentation allows the system to efficiently narrow down the search space without requiring exhaustive manual investigation of all changes, thus reducing time while maintaining accuracy through multiple filtering passes.
Solution Approach 2:
The patent applies partial action by using multiple sorting heuristics that each perform a partial analysis of the code changes. Instead of requiring complete manual review of all changes, the system applies several lightweight sorting passes that collectively identify the most likely culprits. This partial action approach achieves sufficient accuracy without the full time cost of complete manual investigation.
2Measurement precision
If multiple sorting heuristics are applied to code changes, then accuracy in identifying culprit changes improves, but computing cost increases
Solution Approach 1:
The patent uses multiple sorting heuristics that each perform a partial analysis of code changes. Each heuristic is computationally lightweight and processes the change set independently. By using multiple partial analyses rather than one comprehensive expensive analysis, the system achieves higher accuracy through aggregation of results while keeping the computing cost of each individual heuristic low.
Solution Approach 2:
The patent creates multiple sorted lists (copies) of the code changes, each sorted by a different heuristic. Rather than performing one complex analysis, the system creates several simpler sorted versions of the same data set. These multiple copies can be generated independently and in parallel, distributing the computing load while improving identification accuracy through comparison of the sorted results.
3Reliability
If automated test platforms continuously run tests, then code health monitoring is improved, but the ability to provide actionable information about specific failing changes is lost
Solution Approach 1:
The patent implements feedback by taking the output from automated test platforms (test failure information) and processing it through multiple sorting heuristics to generate actionable feedback about specific culprit changes. The system feeds the test results back into the analysis pipeline, applies multiple sorting passes, and produces detailed information about which specific changes are most likely causing failures. This feedback loop transforms raw test failure data into targeted actionable intelligence.
Solution Approach 2:
The patent introduces multiple sorting heuristics as intermediary processing steps between the automated test platform and the developer. The test platform provides raw failure information, and the sorting heuristics act as intermediaries that analyze, sort, and prioritize the code changes to identify culprits. This intermediary layer preserves the continuous monitoring capability while adding the missing actionable information about specific failing changes.
Data Source
AI summary
A system and method for isolating one or more code changes which are suspected of causing a code failure are disclosed. An example system may include a backend, a frontend, and a datastore. A list of changes to a codebase may be received along with a list of test targets in the codebase that are failing and a snapshot of the codebase at the time when one or more of the codebase's tests started failing. A heuristic may be used to find the code changes causing the one or more code failures.