Machine Learning Visual Pairwise Difference Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cross-browser testing and regression testing for event-driven applications are time-consuming and error-prone due to differences in rendering and functionality across various client-tier environments, making it difficult to ensure a high-quality user experience.
Innovation Solution
The implementation of machine learning techniques to improve visual comparison and error prediction by analyzing diverse factors and using a prediction engine to classify pairwise differences in DOM representations across different browsers, reducing false positives and automating the detection of meaningful errors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual cross-browser testing is performed to ensure rendering consistency, then detection precision of visual differences is improved, but productivity is significantly reduced due to time-consuming manual processes
Solution Approach 1:
The patent replaces manual visual inspection (mechanical human process) with an automated image processing system that captures screenshots, computes histograms, and compares color distributions algorithmically. This substitution maintains detection precision while dramatically improving productivity by eliminating time-consuming manual testing.
Solution Approach 2:
The patent introduces histogram computation as an intermediary step between image capture and difference detection. By transforming visual images into histogram representations (intermediary data structure), the system enables automated comparison while preserving the ability to detect subtle rendering differences that manual inspection might miss.
2Productivity
If basic histogram-matching techniques are used to compare images across browsers, then productivity is improved through automation, but measurement precision deteriorates due to false positives from small shifts
Solution Approach 1:
The patent applies preliminary filtering by identifying and excluding known non-critical elements (such as timestamps, session IDs, or dynamically generated content) before performing histogram comparison. This preliminary action prevents false positives from small, insignificant variations while maintaining automation efficiency.
Solution Approach 2:
The patent modifies the comparison parameters by using Earth Movers' Distance (EMD) instead of simple histogram difference metrics. EMD accounts for the spatial distribution and magnitude of color differences, allowing the system to distinguish between significant rendering errors and negligible variations, thereby improving measurement precision while maintaining automated productivity.
3Productivity
If DOM comparison is used to detect differences across browsers, then productivity is improved through automated code analysis, but measurement precision worsens because DOM representations vary significantly between browsers
Solution Approach 1:
The patent creates visual copies (screenshots) of the rendered output rather than comparing the underlying DOM code directly. This copying approach bypasses browser-specific DOM representation variations and focuses on the actual visual rendering, improving measurement precision while maintaining automated productivity through image processing.
Data Source
AI summary
In some embodiments, information associated with a first plurality of image pairs is received. Each image pair is assessed to detect visual pairwise differences that qualify as an error. A visual pairwise difference may be a difference with respect to at least one of position, size, color, or style. A prediction engine is trained based upon the assessed visual pairwise differences. Information associated with a second plurality of image pairs is received. Each of these image pairs comprises at least a portion of a visual end-user experience screen of an event-driven application executed in a client-tier environment. Each of these image pairs is assessed, using the prediction engine, to detect visual pairwise differences that qualify as an error. User feedback is received, indicating that at least one assessed pairwise difference should not have qualified as an error. The prediction engine is then re-trained based on the user feedback.