AI Screen Object Recognition for RPA Beyond Changing Class IDs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional RPA robots struggle to recognize text in images and adapt to changes in Web page class IDs, leading to difficulties in UI recognition and input operations.
Innovation Solution
A method and system using AI technology to detect and recognize RPA input objects on a screen by distinguishing between text and non-text elements, employing AI models like SSD, YOLO, and C-RNN for accurate localization and classification of input objects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional RPA robots use class ID recognition from HTML and JAVASCRIPT sources, then they can recognize search boxes and buttons, but they fail when Web pages change class IDs to address security and RPA
Solution Approach 1:
The patent replaces the mechanical/conventional RPA approach of parsing HTML and JAVASCRIPT class IDs with an AI-based vision system. The AI model processes screenshots of the Web page to recognize UI elements like search boxes and buttons based on their visual appearance and spatial relationships, eliminating dependency on fixed class ID structures that can be changed for security purposes.
Solution Approach 2:
The patent introduces an AI model as an intermediary between the RPA robot and the Web page elements. Instead of directly querying HTML class IDs, the AI model acts as a mediator that translates visual screen information into recognizable UI element positions and characteristics, enabling robust recognition despite class ID changes.
2Extent of automation
If AI learning models are used for screen UI recognition, then event clicks become possible, but text recognition fails in images containing text images
Solution Approach 1:
The patent segments the screen image into different regions and identifies whether each region contains text or non-text UI elements. By dividing the recognition task into text detection and non-text UI detection, the system can appropriately process each type of content, recognizing both text-containing elements and text-only elements without confusion.
Solution Approach 2:
The patent applies different recognition strategies to different local regions of the screen image based on their characteristics. Regions identified as containing text are processed differently from regions with non-text UI elements, allowing the system to accurately recognize both types while avoiding the pitfall of treating all regions uniformly.
3Ease of operation
If existing screen recognition methods are used, then non-text screen UI elements like icons and buttons can be recognized, but text in images such as icon text or application title text cannot be recognized
Solution Approach 1:
The patent creates a universal recognition system that can handle multiple types of screen elements simultaneously - both text-containing UI elements and text-only elements. The AI model is designed to universally process any screen region, adapting its recognition strategy based on whether the region contains text, thereby eliminating the need for separate processing pipelines.
Data Source
AI summary
A method of generating an event for an object on a screen by recognizing screen information based on AI includes accessing a Web-based IT operation management system platform to register a schedule in a scheduler, reporting registration of the schedule to an AI Web Socket of the Web-based IT operation management system platform, transmitting data reporting start of the scheduler from the AI Web Socket of the Web-based IT operation management system platform to an AI Web Socket of an AI screen agent of a PC through communication at a predetermined time, transmitting a PC screen image, and requesting information data, inferring a position of one or more objects on the screen, transmitting information data for the inferred position of the one or more objects, and generating an event for the one or more objects on the PC screen based on the transmitted data.