External GUI Control for Automating Legacy Manufacturing Computers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automation technologies for manufacturing devices, such as Robotic Process Automation (RPA) software, require new computers and struggle to operate on old systems without network connectivity, making it difficult to automate tasks on legacy manufacturing devices due to proprietary control methods and lack of technical materials.
Innovation Solution
A control device that connects externally to a computer, utilizing an image acquisition unit, image processing, and sequence processing to generate control signals from GUI inputs, allowing automation of tasks without analyzing communication protocols, and enabling remote control via a network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If RPA software is installed on old computers to automate tasks, then automation capability is improved, but compatibility and operability deteriorate because RPA software requires new computers with network connectivity
Solution Approach 1:
The patent introduces an external control device as an intermediary between the automation system and the old computer. This control device captures screenshots of the computer's display, processes the visual information to identify GUI elements, and generates control signals to automate interactions. This mediator approach enables automation of legacy systems without modifying the old computer itself or requiring network connectivity, thus resolving the compatibility issue while maintaining automation capability
Solution Approach 2:
The patent replaces traditional programmatic control methods (which require protocol analysis and system integration) with visual-based control. By capturing display images and processing them through image recognition algorithms, the system can identify and control GUI elements without needing to understand the underlying communication protocols or system architecture of the old computer, thereby achieving broad compatibility
2Measurement precision
If communication protocols are analyzed to control the manufacturing device, then control precision is improved, but development time and complexity increase due to proprietary control methods
Solution Approach 1:
The patent creates a visual copy of the device's control interface through screenshot capture. Instead of analyzing and replicating the complex communication protocols and control methods, the system captures the visual state of the display and uses image processing to identify control elements. This copying approach simplifies the control mechanism while maintaining precision, as the system directly interacts with the visual representation of the interface rather than through complex protocol layers
3Productivity
If manual operations are performed on manufacturing devices, then operational flexibility is maintained, but productivity decreases due to repetitive tasks
Solution Approach 1:
The patent implements self-service automation where the control device autonomously performs repetitive operations by capturing display images, identifying GUI elements through image processing, and automatically generating control signals. The system can execute predefined sequences of operations without human intervention, thereby increasing manufacturing throughput while maintaining the simplicity of operation through visual-based control that mirrors manual interaction patterns
Data Source
AI summary
The present disclosure is a control device for automating tasks for control which has been conventionally performed manually for devices including a computer with key input and an user interface (UI), which is difficult to be applied with software such as an RPA that automatically performs tasks in the computer, and the present disclosure is a means for performing automation without analyzing a communication protocol or a control method of a control unit of the computer and a main body of the device, and realizes automation by applying a signal processing technology to a signal of the UI and performing control from an external computer.