Visual Element-Based Application Testing via OCR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing application testing methods face challenges in generating tailored test scripts for different types of code, leading to technical difficulties such as resource utilization overhead and potential failures due to popup notifications, and are limited by the need to analyze accessible code.
Innovation Solution
A visible elements-based application testing approach that uses a GUI-independent method, employing cloud-based analysis tools like MICROSOFT AZURE to identify elements on a page, apply image binarization and OCR, and execute operations based on the presence of test words, regardless of the underlying code, to test applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional code-based testing methods are used, then testing coverage can be achieved, but resource utilization overhead increases and the system becomes limited by code accessibility
Solution Approach 1:
The patent replaces code-based analysis with visual element-based analysis. Instead of parsing and analyzing application code structures, the system captures screenshots and identifies visual elements (buttons, text fields, icons) directly from the application interface. This substitution eliminates the need for code accessibility and reduces processing overhead associated with code analysis.
Solution Approach 2:
The patent creates visual copies of the application interface through screenshots. By working with image representations rather than original code, the system can analyze application behavior without directly accessing or processing the underlying code, thereby reducing resource overhead and removing accessibility constraints.
2Ease of operation
If code analysis is performed for application testing, then testing can be conducted, but the system is limited by the need to access and process internal code structures
Solution Approach 1:
The patent substitutes code-based testing mechanisms with visual-based testing mechanisms. By capturing and analyzing screenshots of the application interface, the system can perform testing operations without needing to access, parse, or process internal code structures, thereby simplifying the testing process and removing complexity associated with code access requirements.
3Productivity
If traditional testing methods are used, then application functionality can be tested, but processing overhead increases due to code analysis requirements
Solution Approach 1:
The patent extracts the essential testing information directly from the visual interface rather than from code. By taking screenshots and identifying visual elements, the system extracts only the necessary information for testing (element locations, text content, interface structure) without the time-consuming process of code parsing and analysis, thereby reducing processing overhead and improving testing efficiency.
Data Source
AI summary
In some examples, visible elements-based application testing may include accessing, based on an analysis of a display of a page associated with execution of an application, an identification of elements and locations of the identified elements within the page. According to an example, each element of the identified elements may include a character. A determination may be made as to whether a word from a test script for testing the application is present in a set that includes at least two of the identified elements by comparing distances between a specified location of the word on the page and the locations of the identified elements. Further, in response to a determination that the word is present in the set of the identified elements, an operation associated with the word may be executed to test the application.