Text Extraction from Application Window Memory for RPA
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Robotic Process Automation (RPA) systems face challenges in accurately understanding and interacting with user interfaces of software applications due to limitations in existing Optical Character Recognition (OCR) methods, which are prone to errors from resolution, size, and display driver issues.
Innovation Solution
Accessing content from process memory used by software programs to retrieve and analyze text presented on user interfaces, allowing RPA systems to accurately understand and interact with application windows through injecting text access program code into the process memory and converting graphical characters to computer-readable text codes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If OCR-based text recognition is used to understand user interface content, then the RPA system can interact with applications without integration, but the text recognition accuracy deteriorates due to resolution, size, and display driver issues
Solution Approach 1:
The patent introduces an intermediary text access program code that acts as a mediator between the RPA system and the application's process memory. This code injects into the target application's memory space and retrieves text directly from the application's internal data structures, thereby avoiding the inaccuracies of OCR while maintaining the non-integrated interaction model. The intermediary layer translates application memory content into readable text without requiring visual capture.
Solution Approach 2:
The patent replaces the mechanical/optical OCR process with a direct memory access mechanism. Instead of capturing screen images and performing optical character recognition, the system substitutes this with programmatic injection of text access code into the application's process memory, where text is retrieved directly from memory structures. This substitution eliminates the resolution and display driver dependencies inherent in OCR-based approaches.
2Loss of information
If screen capture with OCR is used to extract text from user interfaces, then the RPA system can access application content, but the reliability deteriorates due to errors from resolution and display driver variations
Solution Approach 1:
The text access program code serves as an intermediary that directly interfaces with the application's process memory. This intermediary retrieves text data from the application's internal memory structures before it is rendered to the display, eliminating the need for screen capture and subsequent OCR processing. This direct access path ensures reliable text extraction independent of display resolution, drivers, or rendering variations.
Solution Approach 2:
The system performs preliminary action by injecting the text access program code into the application's process memory before text extraction is needed. This pre-positioned code establishes a direct communication channel with the application's memory space, allowing reliable text retrieval without relying on visual capture at the time of extraction. The preliminary injection ensures the text access mechanism is in place and ready, avoiding reliability issues associated with on-the-fly screen capture.
3Loss of information
If text is accessed through screen capture and OCR, then the RPA system can understand user interface content, but the process complexity increases due to multiple processing steps and error handling requirements
Solution Approach 1:
The patent extracts the text access functionality directly into the application's process memory through code injection. By taking out the text retrieval operation from the external OCR processing pipeline and embedding it within the target application's memory space, the system eliminates multiple processing steps including screen capture, image preprocessing, OCR engine invocation, and result validation. This extraction simplifies the overall process to a single direct memory access operation.
Solution Approach 2:
The injected text access program code acts as a streamlined intermediary that directly bridges the RPA system and application content. This single intermediary layer replaces the complex multi-step OCR pipeline, reducing process complexity while maintaining the ability to understand user interface content. The intermediary handles all text extraction operations through a unified memory access interface, eliminating the need for separate error handling for capture failures, OCR recognition errors, and display variations.
Data Source
AI summary
Improved techniques to access content presented on a user interface by a software program are disclosed. The improved techniques permit the content to be retrieved from process memory used by the software program. Once accessed, the content (e.g., text) can be analyzed and used to understand and interact with the software program (e.g., application program) by way of the user interface (e.g., application window). The improved techniques are well suited for Robotic process automation (RPA) system. In such cases, RPA systems are able to accurately understand the content of the user interfaces such that software automation processes can operate with increased reliability and flexibility.