Visual Software Test Validation Using DOM-Screenshot Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Pixel-based methods for visually validating web pages or apps fail to accurately identify web elements due to changes in text content or visual elements split across multiple pixels, leading to misalignment and incorrect validation.
Innovation Solution
A method that combines document object model (DOM) information with screenshot analysis to identify web page elements, allowing user input to correct element selection and reinforce the model through neural networks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If pixel-based methods are used to visually validate web pages, then visual comparison can be performed, but text content changes cause misalignment and validation failures
Solution Approach 1:
The patent introduces DOM information as an intermediary layer between the screenshot and the validation process. Instead of directly comparing pixels, the system uses DOM data to identify and locate web elements, then performs validation based on element properties rather than raw pixel values. This mediator resolves the conflict between visual fidelity and text content variability.
Solution Approach 2:
The patent transitions from a two-dimensional pixel-based validation approach to a hybrid approach that incorporates the DOM tree structure as an additional dimension. By mapping screenshot regions to DOM elements and using element hierarchy for validation, the system adds structural context that helps maintain reliability despite visual variations.
2Difficulty of detecting and measuring
If pixel-based methods segment images into elements, then visual boundaries can be detected, but elements without direct visual correspondence or split across multiple elements cannot be identified
Solution Approach 1:
The patent merges two complementary approaches: visual segmentation (using edge detection and feature recognition) and DOM-based element identification. The visual method detects potential element boundaries, while the DOM information provides the ground truth element structure. By combining these sources, the system overcomes the limitations of each individual approach.
Solution Approach 2:
The system uses DOM information as feedback to correct and refine visual segmentation results. When visual methods detect boundaries that don't correspond to actual DOM elements, or miss elements that lack clear visual boundaries, the DOM structure provides corrective feedback to identify the correct element mappings.
3Device complexity
If traditional visual validation is used, then implementation is simple, but maintenance time increases as web pages change
Solution Approach 1:
The patent enables the test validation system to self-adapt to web page changes by automatically incorporating DOM information. When the DOM structure changes, the system can automatically update its element mappings and validation criteria without requiring manual reconfiguration, significantly reducing maintenance time while keeping the core validation logic relatively simple.
Data Source
AI summary
Provided herein is technology relating to testing software and particularly, but not exclusively, to systems and methods for visually validating a step in a software test case and using the visual validation to reinforce an element selection model, the technology provides methods for visually validating a step in a software test case. More particularly, embodiments of the technology provide a method of mapping web elements of a website or app to a visual screenshot of that website or app by combining data from the document object model (DOM) and page screenshot.