Image Recognition Auto-Fix for Web Test Object Errors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional software testing methods relying on the Document Object Model (DOM) are overly sensitive to changes in the DOM structure, leading to frequent 'object not found' errors, especially when testing third-party applications or when legacy scripts are incompatible with updated DOMs, requiring manual updates that are time-consuming and can break automation processes.
Innovation Solution
The use of image recognition and Artificial Intelligence to identify graphical objects on a web page, capturing screenshots and using text syntax to locate objects based on visual attributes, allowing for automatic updating of test scripts without relying on the DOM structure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Difficulty of detecting and measuring
If DOM-based methods are used to find objects in web pages, then object finding can be performed through JavaScript code execution, but the methods become overly sensitive to DOM changes causing frequent object not found errors
Solution Approach 1:
The patent replaces the mechanical/DOM-based object finding system with an optical/image recognition system. Instead of using JavaScript to parse DOM structure, the system captures screenshots and uses image recognition algorithms to locate objects visually, substituting the traditional programming-based approach with a perception-based approach that is invariant to DOM changes
Solution Approach 2:
The patent creates a visual copy (screenshot) of the web page instead of relying on the original DOM structure. By working with the visual representation rather than the underlying code structure, the system can identify objects based on their visual appearance which remains consistent even when DOM attributes change
2Reliability
If manual updates are performed to fix object not found errors, then test scripts can be corrected, but the process is time-consuming and breaks automation
Solution Approach 1:
The system performs self-service by automatically detecting object not found errors and autonomously fixing them through image recognition. Instead of requiring manual intervention, the system independently identifies missing objects, determines their locations through visual analysis, and updates test scripts automatically, making the testing process self-healing
Solution Approach 2:
The patent implements a feedback mechanism where the system monitors test execution, detects object not found errors, uses image recognition to analyze the current page state, determines the correct object locations, and feeds this information back to automatically update the test scripts. This closed-loop system continuously improves itself without human intervention
3Extent of automation
If legacy test scripts are used with updated DOM structures, then existing automation can be maintained, but compatibility issues arise causing object not found errors
Solution Approach 1:
The patent segments the object identification process into two independent parts: the legacy test script (which remains unchanged) and the image recognition system (which adapts to new DOM structures). By separating the testing logic from the object identification mechanism, the system allows legacy scripts to continue working while the visual recognition layer adapts to DOM changes
Data Source
AI summary
A system, device, system-on-a-chip, and method of automatically correcting an object not found error using image recognition are described. The method includes running a test script for testing and analysis of a web page as rendered by a web browser. The method further includes, responsive to detecting the object not found error, automatically locating a missing object associated with the object not found error. One method of locating a missing object includes using image recognition. The method also includes updating the test script with a located object. The method may also include replaying the test script.