Graphical Object Identification via Text Syntax Parameters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current software testing methods using image analysis face challenges in accurately identifying graphical objects in a user interface due to unnecessary information in images, which reduces the accuracy of image comparison algorithms.
Innovation Solution
The system captures an image of a graphical user interface and uses text syntax with parameters to identify specific graphical objects, returning coordinates for accessing these objects, which can be used in test scripts to interact with the interface.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If image comparison algorithms are used to identify graphical objects, then the system can process full image data, but the accuracy is reduced due to unnecessary information such as additional image edges, unimportant colors, and shapes
Solution Approach 1:
The patent segments the graphical user interface into distinct graphical objects by detecting edges and grouping pixels into object regions. This segmentation separates relevant graphical objects from unnecessary background information, enabling accurate identification while reducing processing complexity by focusing only on object regions rather than the entire image.
Solution Approach 2:
The patent extracts essential features (such as object boundaries, colors, and shapes) from the full image data while discarding unnecessary information. By taking out only the relevant graphical object characteristics needed for identification, the system achieves high accuracy without being burdened by extraneous image data.
2Measurement precision
If text syntax with parameters is used to identify graphical objects, then the accuracy of identification is enhanced by filtering unnecessary information, but the complexity of the identification system increases
Solution Approach 1:
The patent uses text syntax with parameters (such as object type, color, position, and text content) to define and identify graphical objects. By changing the representation from raw image data to parameterized object descriptions, the system achieves precise identification while making the complexity manageable through structured, queryable parameters.
3Adaptability or versatility
If captured images are used for software testing, then access to source code is not required, but the accuracy of identifying specific graphical objects is reduced due to unnecessary image information
Solution Approach 1:
The patent introduces an intermediary processing layer that captures images of the graphical user interface and transforms them into structured graphical object data with parameters. This intermediary layer enables source code-independent testing while improving identification accuracy by filtering and structuring the captured image information into meaningful object representations.
Data Source
AI summary
An image of a graphical user interface is captured. For example, a screen shot of a browser display is captured. Text syntax is executed that contains one or more parameters for identifying a graphical object. For example, the text syntax may identify a rectangle that contains the text “OK” where the text is red. Based on the text syntax, a graphical object is identified in the image of the graphical user interface. Information is returned that identifies how to access the graphical object in the graphical user interface. For example, coordinates of the graphical object are identified. This information can then be used in a test script using existing programming languages to test the graphical user interface. For example, the coordinates may be used to click on the OK button.