GUI Image Analysis for Remote Application Interoperability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in enabling graphic-based interoperability between computer systems, particularly in remote environments like Microsoft Remote Desktop, Citrix XenApp, and Oracle VM, due to the lack of reliable connectors and object models for GUI elements, limiting interaction and integration with applications.
Innovation Solution
A system and method that analyze graphical images of GUIs to identify changes, update object constructs, and raise events, enabling communication and integration between applications without requiring direct API access, using shape analysis and image processing algorithms to recognize and interact with GUI elements in real-time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If standard API and SDK techniques are used to connect applications, then interoperability is achieved for applications with exposed interfaces, but applications executing in remote environments cannot be connected or have their internal events easily known
Solution Approach 1:
The patent introduces an intermediary component that captures graphical images of the remote application interface and processes them to extract control element information. This intermediary approach allows connection to remote applications without requiring direct API access, bridging the gap between local and remote environments.
Solution Approach 2:
The patent creates a graphical copy or representation of the remote application interface by capturing its visual output. This copy is then analyzed to identify control elements, allowing interaction with the remote application through its visual representation rather than requiring direct access to the original application's internal structures.
2Adaptability or versatility
If graphical images of GUIs are analyzed to identify changes and interact with control elements, then interoperability with remote applications is enabled, but the complexity of the system increases due to image processing requirements
Solution Approach 1:
The patent divides the graphical image analysis into distinct segments: capturing the graphical image, identifying control elements within the image, determining their properties, and mapping them to corresponding application events. This segmentation simplifies the overall complex task of graphical interface analysis.
Solution Approach 2:
The patent replaces traditional mechanical or direct-programmatic interaction mechanisms with image-based recognition and analysis. Instead of using direct API calls or SDK interfaces, the system uses graphical image processing to identify and interact with control elements, substituting visual pattern recognition for conventional programmatic access.
3Speed
If image processing algorithms are used to recognize GUI elements in real-time, then real-time interaction is achieved, but the processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary analysis of the graphical image to identify control elements and their properties before actual interaction is needed. This pre-processing allows the system to have control element information ready in advance, reducing the computational burden during real-time interaction operations.
Solution Approach 2:
The patent focuses image processing efforts on specific regions or control elements that are relevant to the current interaction context, rather than analyzing the entire graphical interface uniformly. This partial action approach reduces overall processing requirements while maintaining real-time interaction capability for critical elements.
Data Source
AI summary
A system and method for enabling graphic-based interoperability between computer executed applications. A computer system operating as a client may display a graphical user interface (GUI) including control graphic items such as buttons, text boxes, etc. A process may examine the graphical image of the GUI to determine if there has been a change over time in the GUI as displayed which updates a control graphic item. If there has been a change over time in the GUI which updates a control graphic item, an action may be taken, for example updating properties of an object construct corresponding to the control graphic item, raising an event corresponding to an object construct corresponding to the control graphic item, or communicating an event to a process.