Graphic-Based Interoperability for Remote Desktop Table Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in achieving interoperability between computer applications, particularly in remote environments, due to the lack of reliable connectors and object models for GUI elements, which hinders interaction with applications like those in Microsoft Remote Desktop, Citrix XenApp, and Oracle VM systems, especially for dynamic tables that change over time.
Innovation Solution
A system and method for graphic-based interoperability that allows accessing elements in a digital image of a table by obtaining the digital image, identifying table elements, calculating their coordinates, and interacting with the table content, using methods such as orthogonality filter, gridlines detection, morphology detection, and cartoonizing to extract table properties and elements, enabling communication and integration of applications independent of platforms or operating systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If standard API and SDK techniques are used to connect applications, then interoperability between applications can be achieved, but these techniques cannot recognize UI elements or connect to APIs in remote environments where only digital images of applications are available
Solution Approach 1:
The patent introduces an intermediary system that captures digital images of remote application interfaces and processes them to extract UI element information. This intermediary acts as a bridge between the remote application (which exposes only visual interface) and the local application (which needs to interact with it), enabling interoperability without direct API access to the remote system.
Solution Approach 2:
The patent replaces traditional mechanical/API-based connection methods with image processing-based interaction. Instead of using standard API calls that require direct system access, the system captures visual representations of the interface, processes the images to identify UI elements, and simulates user interactions based on the extracted visual information.
2Loss of information
If OCR engines are used to recognize tables in documents, then text can be extracted and formatted, but these solutions cannot support real-time interaction with dynamic tables or enable changing table content
Solution Approach 1:
The patent implements dynamic table recognition by continuously capturing images of the table interface and comparing them across time intervals. The system detects changes in table structure and content by analyzing differences between sequential images, enabling real-time tracking of dynamic table updates and facilitating interactive operations on current table states.
Solution Approach 2:
The patent establishes continuous image capture and processing cycles to maintain an up-to-date representation of the table interface. By continuously monitoring the visual interface and processing images in real-time, the system ensures that table content extraction and interaction operations always work with the current state of the table, enabling productive real-time operations.
3Ease of operation
If applications are installed on remote servers to enable interaction, then full functionality can be accessed, but this increases software installation requirements and system complexity
Solution Approach 1:
The patent creates a visual copy of the remote application interface by capturing digital images of the interface elements. Instead of installing the actual application on the remote server, the system captures images of the interface, processes them to extract UI element information, and uses these processed images to enable interaction. This copying approach eliminates installation requirements while maintaining interaction capability.
Data Source
AI summary
A system and method for accessing elements of a table in a digital image of the table, including: obtaining the digital image of the table; finding table elements in the digital image based on digital table properties, wherein the table elements define table cells; calculating coordinates of the table cells in the digital image based on the table elements; and accessing content of a selected table cell in the digital image using the coordinates of the selected table element.