Display Artifact Recognition Without UI APIs or Layout Knowledge
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies cannot efficiently extract data from graphical displays without prior knowledge of the user interface or an application programming interface, making it difficult for separate applications to interact with displayed artifacts such as radio buttons and text fields.
Innovation Solution
A processor identifies and processes graphical display data to recognize alpha-numeric and non-alpha-numeric artifacts by defining regions, enhancing boundaries, and linking them, even without prior knowledge of the display layout, using techniques like edge detection, geometric analysis, and optical character recognition to facilitate interaction with these artifacts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If anti-aliasing is applied to smooth text for human consumption, then readability is improved, but artifact recognition accuracy deteriorates
Solution Approach 1:
The patent segments the anti-aliased text into individual characters by detecting vertical projection profiles and identifying character boundaries. This segmentation allows the system to process text as discrete recognizable units, enabling accurate recognition despite the smoothing effects of anti-aliasing.
Solution Approach 2:
The patent applies preliminary processing steps including converting to grayscale, applying vertical projection profiles, and identifying character boundaries before actual recognition. These preliminary actions prepare the anti-aliased text for accurate recognition by pre-segmenting and positioning character regions.
2Adaptability or versatility
If no API is provided by the first application, then application independence is improved, but data access capability deteriorates
Solution Approach 1:
The patent introduces an intermediary system that captures images of the display and uses image processing techniques to extract data. This intermediary layer enables the second application to access data from the first application's display without requiring direct API access, maintaining application independence while enabling data sharing.
Solution Approach 2:
The patent creates a visual copy of the display content through image capture and processing. By working with a copy of the display image rather than direct data access, the system enables data extraction without requiring the first application to expose its internal data structures or provide APIs.
3Adaptability or versatility
If no prior knowledge of UI layout is available, then system versatility is improved, but artifact location accuracy deteriorates
Solution Approach 1:
The patent enables the system to automatically analyze and interpret the display layout without requiring pre-programmed knowledge of specific UI structures. The system self-adapts to any display content by using general image processing techniques such as edge detection, projection profiles, and pattern recognition to locate artifacts.
Solution Approach 2:
The patent changes the approach from using fixed UI model parameters to using dynamic image analysis parameters. By analyzing the actual visual properties of the display (edge patterns, projection profiles, color distributions), the system adapts its artifact location accuracy to any given display layout without requiring prior knowledge.
Data Source
Figure 1~2
Figure 3~4
Figure 5~6
AI summary
A digital display of an active window is provided. An application, separate from the application providing the digital display, is capable of identifying an artifact in the active window of the digital display without a priori knowledge or an application programming interface to the application providing the digital display. The artifacts may be alpha-numeric artifacts or non-alpha-numeric artifacts. The active window may be examined in multiple dimensions including 2 dimensions, 3 dimensions or more. Alpha-numeric artifacts, once identified, are pre-processed to remove anti-aliasing to allow for processor optical recognition.