Automated Test Failure Analysis Algorithm
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Service automation systems face inefficiencies in analyzing large numbers of automated tests, where testers spend excessive time determining the causes of failed tests, often reviewing redundant results due to similar or previously unaddressed issues, leading to a time-consuming and inefficient process.
Innovation Solution
An algorithm is implemented to detect redundant test failures by analyzing attributes such as messages, stack traces, and screenshots, which are stored in a knowledge base and compared to previous failures, using weight coefficients and similarity metrics like Damerau-Levenshtein distance and Dice coefficient to identify redundant causes and discard them from further analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If testers manually review all failed test results to determine causes, then comprehensive analysis is achieved, but time consumption increases significantly
Solution Approach 1:
The patent introduces an intermediary system consisting of a knowledge base and automated analysis algorithm that mediates between the large volume of failed test results and the tester. The knowledge base stores historical failure data and patterns, while the algorithm automatically compares new failures against this stored knowledge, filtering out redundant cases before presenting only unique failures to the tester for manual review.
Solution Approach 2:
The system enables self-service by allowing the automated test framework to autonomously analyze and categorize its own failure results. The algorithm automatically identifies redundant failures by comparing attributes like error messages, stack traces, and screenshots against the knowledge base, and self-manages the update of the knowledge base with newly identified unique failure patterns, reducing the need for continuous manual intervention.
2Reliability
If all failed tests are analyzed in detail, then all potential bugs are detected, but redundant analysis of similar failures increases workload
Solution Approach 1:
The patent extracts and separates redundant failure cases from the set of all failed tests by comparing their attributes (error messages, stack traces, screenshots) against stored patterns in the knowledge base. Only the essential unique failures that represent new bug types are extracted for detailed analysis, while redundant instances are automatically identified and set aside, allowing testers to focus on novel issues without repeating analysis of known problems.
3Measurement precision
If the knowledge base is continuously updated with new failure patterns, then detection accuracy improves, but system complexity increases
Solution Approach 1:
The system performs preliminary actions by pre-processing and storing failure patterns in the knowledge base before they are needed for comparison. The knowledge base is proactively updated with new failure patterns as they are discovered, and the algorithm is pre-configured with comparison criteria and attributes. This preliminary organization of data structures and comparison logic simplifies the real-time analysis process, as the system already has structured data ready for efficient matching rather than having to build analysis frameworks during each test run.
Data Source
AI summary
A large number of tests are performed on a service automation system. An algorithm for detecting redundant test failures is provided. Each failed automated test has a set of attributes that allow for determining a degree of similarity with previous failed tests. Each failed test generates an exception containing a message and a stack trace. A message, a stack trace, a screenshot and other attributes produced by the test at the point of failure are included into a knowledge base of failed tests. This data can be analyzed for determining a cause of the test failures. The actual cause, once determined, is saved into a database and linked to a particular test failure. In order to find the redundant test failures, the attributes of a current test failure are compared to corresponding attributes of the previous test failures. The redundant test results and their causes are discarded from further test analysis.