Image-Based Automation for Cross-Platform Software Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional software automation systems rely on understanding and communicating with the runtime environment of software applications, limiting their ability to automate applications across different platforms and requiring modifications when runtime environments change.
Innovation Solution
Image-based automation systems identify and record actions using visual representations of graphical objects, allowing them to automate software applications without relying on runtime environments, enabling automation across various platforms without the need for modifications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional software automation systems communicate with the runtime environment to receive information and send instructions, then they can perform automated actions within that specific runtime environment, but they cannot automate applications across different platforms and require modifications when runtime environments change
Solution Approach 1:
The patent introduces an image processing module as an intermediary between the automation system and the software application. Instead of directly communicating with the runtime environment, the system captures screenshots, processes them to identify graphical objects, and performs actions based on visual information. This intermediary layer enables cross-platform automation without requiring direct integration with different runtime environments.
Solution Approach 2:
The patent replaces the traditional mechanical approach of direct runtime environment communication with an optical/image-based approach. By substituting the direct software-to-software communication mechanism with screenshot capture and image processing, the system achieves platform independence while maintaining automation capabilities.
2Reliability
If software automation systems rely on understanding the runtime environment, then they can accurately control actions within that environment, but they require modifications when the runtime environment changes
Solution Approach 1:
The patent creates visual copies (screenshots) of the runtime environment and processes these copies to identify graphical objects and determine actions. By working with image copies rather than direct runtime environment interactions, the system maintains action accuracy while becoming independent of runtime environment changes, eliminating the need for modifications when environments update.
3Productivity
If traditional automation systems use record and replay paradigm with runtime environment communication, then they can automate sequences of actions, but they lack flexibility when applications are updated or moved between platforms
Solution Approach 1:
The patent substitutes the runtime environment communication mechanism with image capture and processing. This allows the record and replay paradigm to function across different platforms and application versions, maintaining productivity while achieving environment independence through visual information rather than direct environment interaction.
Data Source
AI summary
In one implementation, an image-based automation process includes identifying a graphical object of a graphical user interface and performing an action relative to the graphical object at the graphical user interface. The identifying is based on an image including the graphical object.