Software Robot Object Hierarchy for GUI Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional techniques for programmatically controlling computer programs through their graphical user interfaces are inefficient due to high computational costs in determining and updating the state of the GUI, limiting their practicality and ability to control multiple applications across different GUI technologies and virtual machines.
Innovation Solution
The development of software robots that utilize an object hierarchy to represent and efficiently manage the state of GUI elements, allowing real-time control of computer programs across various GUI technologies and virtual machines, with a unified object hierarchy providing a common interface for controlling similar GUI elements across different technologies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional techniques are used to programmatically control computer programs through their graphical user interfaces, then control capability is provided, but computational costs become excessively high due to the need to determine and update the state of the GUI
Solution Approach 1:
The system segments the GUI control process by introducing an object hierarchy that divides the GUI into discrete, manageable objects. Each GUI element is represented as a distinct object in the hierarchy, allowing the software robot to interact with specific elements without needing to analyze the entire GUI state, thereby reducing computational overhead while maintaining control capability.
Solution Approach 2:
The patent introduces an object hierarchy as an intermediary layer between the software robot and the actual GUI. This hierarchy acts as a mediator that translates high-level control commands into low-level GUI interactions, eliminating the need for the software robot to directly compute and update the complete GUI state, thus significantly reducing computational costs.
2Adaptability or versatility
If conventional techniques are used to control multiple applications across different GUI technologies, then versatility is improved, but device complexity increases due to the need to handle multiple APIs and technologies
Solution Approach 1:
The object hierarchy provides a universal interface that can represent and control GUI elements across different technologies and applications. By defining a standardized hierarchy structure that works with various GUI frameworks, the system achieves multi-functionality without requiring separate control mechanisms for each technology, thereby reducing overall system complexity while maintaining versatility.
Solution Approach 2:
The object hierarchy serves as a technology-agnostic intermediary layer that sits between the software robot and diverse GUI technologies. This mediator translates unified control commands into technology-specific operations, allowing the system to control multiple applications across different GUI technologies without exposing the complexity of individual APIs to the control logic.
3Measurement precision
If developers directly interact with low-level APIs to control GUI elements, then precise control is achieved, but ease of operation deteriorates due to the complexity of managing low-level details
Solution Approach 1:
The system creates an object-based representation (a copy) of the GUI hierarchy that mirrors the actual GUI structure. Developers interact with this simplified object model rather than the complex low-level APIs, maintaining precise control over GUI elements while significantly improving ease of operation. The object hierarchy preserves the hierarchical relationships and properties of actual GUI elements, enabling accurate control without direct API complexity.
Data Source
Figure 1A~1B
Figure 2A~2B
Figure 3A~3B
AI summary
Techniques for developing, deploying, and using software robot computer programs for programmatically controlling one or more other computer program(s) to perform a task via an object hierarchy that provides a representation of graphical user interface (GUI) elements of the computer program(s) being controlled. The object hierarchy may include objects corresponding to active GUI elements of the computer program(s) being controlled. The software robot may be configured to generate the object hierarchy, refresh the object hierarchy, and search for objects in the object hierarchy. A software robot may be configured to control multiple computer programs executing on a same device, on different devices, on a same virtual machine, and/or on multiple virtual machines. The multiple computer programs may be implemented using the same or different GUI technologies.