Auto-triage A/B Testing Failures via Image Deviation Grouping

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Multi-tiered systems require significant testing efforts due to disparate testing tools and techniques, making it challenging to identify software regressions and triage failures efficiently, especially in A/B testing scenarios where frontend and backend components interact.

Innovation Solution

A method for auto-triaging failures involves obtaining screenshots of user interface outputs from different software versions, identifying image deviations, grouping them, and associating failures to determine root causes, thereby automating the grouping and analysis of similar failures.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If manual testing and analysis methods are used to identify software regressions in multi-tiered systems, then testing coverage can be comprehensive, but the time and resource consumption increases significantly

Engineering Contradiction:

Improvesoftware regression detection accuracyVSAvoidtesting cycle duration

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent uses screenshot images as visual copies of the user interface outputs to represent software behavior. Instead of analyzing actual software execution, the system captures and compares visual representations (screenshots) of different software versions, enabling efficient regression detection through image deviation analysis rather than time-consuming manual testing

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The patent replaces manual mechanical testing processes with automated image processing and comparison algorithms. The system automatically captures screenshots, identifies deviations between versions, and groups failures without human intervention, substituting the mechanical manual analysis process with automated computational image analysis

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Reliability

If comprehensive testing of multiple software versions is performed, then software reliability improves, but computer resource usage increases

Engineering Contradiction:

Improvesoftware quality assuranceVSAvoidcomputer resource consumption

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The patent extracts only the essential visual information needed for regression detection by capturing screenshots of specific user interface outputs. Instead of analyzing entire software systems or all possible test cases, the system extracts and compares only the relevant visual portions that indicate software behavior changes, reducing computational resource requirements while maintaining testing effectiveness

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent changes the testing approach from code-level or functional parameter analysis to visual parameter comparison. By transforming software output into image format and comparing visual parameters (pixel values, layout positions, element appearances), the system achieves comprehensive testing with lower resource consumption through efficient image processing algorithms

Inventive Principle:

Principle #35Parameter changes

3Measurement precision

If detailed analysis of each failure is performed manually, then root cause identification is accurate, but the complexity of the testing process increases

Engineering Contradiction:

Improvefailure root cause identificationVSAvoidtesting process complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent merges multiple individual failure analyses by grouping image deviations that share common characteristics. Instead of analyzing each failure separately, the system clusters similar deviations together, identifying patterns across multiple test cases. This consolidation maintains accurate root cause identification while simplifying the overall testing process through automated pattern recognition

Inventive Principle:

Principle #5Merging (Combining)

4Productivity

If automated image comparison is implemented, then testing productivity increases, but the precision of deviation detection may decrease

Engineering Contradiction:

Improvetesting throughputVSAvoidimage deviation detection accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The patent implements feedback mechanisms where the automated image comparison system continuously refines its deviation detection based on identified patterns. The system analyzes grouped deviations, learns from previous comparisons, and adjusts its detection sensitivity accordingly. This feedback loop maintains high detection precision while preserving the benefits of automated high-speed processing

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS12099575B2Auto-triage failures in A/B testing

Publication Date: 2024.09.24 THOUGHTSPOT INC

AI summary

First images that are screenshots from a first version of a software component are obtained. Second images that are screenshots from a second version are obtained. A collection of image deviations that includes pair-wise image deviations between pairs of images are identified. A pair of images includes a first image from the first images and a corresponding second image from the second images. An image deviation indicates a portion of the second image identified as differing from a spatially corresponding portion of the first image. The image deviations are grouped into deviation groups. At least some of the second images are associated with at least some of the deviation groups. A subset of the second images corresponding to a deviation group is output responsive to a selection of an indication of the deviation group.