Synthetic Data Training for OCR Text Detection in Virtual Desktop Environments
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current robotic process automation (RPA) techniques face challenges in accurately recognizing text in images, particularly in virtual desktop environments (VDEs) due to the lack of selectors in image frames, leading to insufficient reliability and high error rates in optical character recognition (OCR) models.
Innovation Solution
The development of a computer program that generates synthetic data to train OCR models using a combination of Faster R-CNN and ResNet architectures, with a single LSTM layer for text recognition, and employs human-labeled data augmentation to improve accuracy, allowing for robust text detection and recognition in RPA systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional OCR techniques are used for text recognition in VDE images, then the system can process images without selectors, but the detection accuracy and reliability are insufficient
Solution Approach 1:
The patent applies preliminary action by pre-training the OCR model on synthetic data before deploying it to real VDE images. The system generates synthetic images with text elements and their corresponding bounding boxes in advance, allowing the model to learn text detection patterns before encountering actual VDE screenshots where selectors are unavailable. This pre-training phase prepares the model to reliably detect and recognize text in the target environment.
Solution Approach 2:
The patent uses copying by creating synthetic images that replicate the visual characteristics of VDE screenshots. These synthetic images contain text elements, buttons, and interface components that mimic real application interfaces, allowing the model to learn from copied representations without requiring actual labeled VDE images. The synthetic data copies the essential features needed for text detection while avoiding the scarcity of real labeled data.
2Measurement precision
If more human-labeled data is collected for training, then model accuracy improves, but the time and cost for data collection and labeling increase significantly
Solution Approach 1:
The patent replaces the need for extensive human-labeled data with synthetic data copies. Instead of manually collecting and labeling hundreds or thousands of real VDE screenshots, the system generates synthetic images programmatically with automatically generated text and bounding box annotations. This copying approach maintains training effectiveness while eliminating the time-consuming manual labeling process.
Solution Approach 2:
The system applies self-service by automatically generating its own training data without requiring human annotators. The synthetic data generation process automatically creates images, inserts text elements, and generates corresponding bounding box annotations through programmatic means. This self-service data generation eliminates dependency on external human labeling resources while providing sufficient training data for accurate text recognition.
3Extent of automation
If traditional selector-based RPA methods are used, then automation can be achieved in native applications, but the method fails completely in VDE environments where selectors are not available
Solution Approach 1:
The patent achieves universality by making the OCR model capable of functioning across multiple environments - both native applications and VDE screenshots. The model learns to detect text and interface elements in a unified manner that works regardless of whether the input is a native application window or a VDE screenshot. This multi-functional capability allows the same automation approach to work across different deployment scenarios without environment-specific modifications.
Solution Approach 2:
The patent substitutes the mechanical selector-based system with an image-based OCR system. Instead of relying on the mechanical property of selector identification that works only in native applications, the system uses visual pattern recognition through OCR that can process any image input. This substitution replaces the environment-dependent selector mechanism with a universal image processing approach that functions in both native and VDE contexts.
Data Source
AI summary
Techniques for training an optical character recognition (OCR) model to detect and recognize text in images for robotic process automation (RPA) are disclosed. A text detection model and a text recognition model may be trained separately and then combined to produce the OCR model. Synthetic data and a smaller amount of real, human-labeled data may be used for training to increase the speed and accuracy with which the OCR text detection model and the text recognition model can be trained. After the OCR model has been trained, a workflow may be generated that includes an activity calling the OCR model, and a robot implementing the workflow may be generated and deployed.