AI Screen Agent for RPA Object Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional RPA robots face challenges in recognizing and interacting with web pages due to changing HTML class IDs and are limited to specific environments like Web browsers, making it difficult to operate in remote terminals and non-Windows OS, such as IoT devices.
Innovation Solution
A method and system using AI technology to infer screen content and generate events for objects on a display screen by training an AI model with screen images, allowing recognition and interaction across various devices and environments without relying on specific APIs or anchors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional RPA robots use learned class IDs from HTML and JAVASCRIPT sources to recognize web page elements, then recognition and input operations can be performed in stable environments, but the system fails when HTML class IDs change, making recognition impossible
Solution Approach 1:
The patent replaces the mechanical system of parsing HTML code and matching class IDs with an AI-based image recognition system. The RPA robot captures screenshots of the web page and uses trained neural network models to identify elements visually, bypassing the need for HTML structure analysis. This substitution allows the system to recognize elements based on their visual appearance rather than relying on potentially changing HTML class attributes.
Solution Approach 2:
The patent creates visual copies (screenshots) of the web page interface and uses these image copies for element recognition instead of working directly with the HTML code. By training AI models on labeled screenshots with annotated element positions, the system learns to identify elements from visual representations, making it resilient to HTML structure changes while maintaining reliable recognition.
2Reliability
If RPA robots are designed to operate in specific environments like Web browsers with specific APIs, then stable operation is achieved in those environments, but the system cannot operate in remote terminals or non-Windows OS such as IoT devices
Solution Approach 1:
The patent implements a universal RPA robot design that can operate across multiple platforms and environments. By using AI-based screen capture and image recognition instead of platform-specific APIs, the same RPA system can function on Windows, macOS, Linux, remote terminals via RDP, and even on IoT devices with different operating systems. The core functionality remains consistent while adapting to different environmental constraints.
Solution Approach 2:
The patent introduces an intermediary layer consisting of screen capture functionality and AI image recognition models that mediates between the RPA robot and the underlying operating system or terminal environment. This intermediary approach allows the robot to interact with the graphical interface universally without needing to understand or rely on specific OS APIs, enabling cross-platform operation while maintaining operational stability.
Data Source
AI summary
A method of generating an event for an object on a screen by recognizing screen information based on AI includes accessing a Web-based IT operation management system platform to register a schedule in a scheduler, reporting registration of the schedule to an AI Web Socket of the Web-based IT operation management system platform, transmitting data reporting start of the scheduler from the AI Web Socket of the Web-based IT operation management system platform to an AI Web Socket of an AI screen agent of the user PC through communication at a predetermined time, transmitting a user PC screen image, and requesting information data, inferring a position of one or more objects on the screen, transmitting information data for the inferred position of the one or more objects, and generating an event for the one or more objects on the user PC screen based on the transmitted data.