CV Model Retraining for RPA Interface Recognition Errors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current CV models for robotic process automation (RPA) struggle with misidentifying or failing to identify graphical components, particularly in virtual machine environments, leading to suboptimal performance and the need for retraining due to interface changes, which is often inefficient and user-dependent.
Innovation Solution
A system that captures screenshots of graphical interfaces, uses a CV model with OCR to identify components, allows users to select misidentified components, and employs a multi-anchor matching algorithm to retrain the model, using deep learning neural networks for accurate component recognition.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If users manually select and send screenshots of misidentified components to the service provider for retraining, then the CV model can be retrained to improve identification accuracy, but the user effort and time required increases significantly
Solution Approach 1:
The system automatically captures screenshots, identifies misidentified components using the CV model, and prepares training data without requiring user intervention. The automation workflow self-identifies errors and self-prepares retraining datasets, eliminating the need for users to manually select and send error reports.
Solution Approach 2:
The system implements an automated feedback loop where the CV model's identification results are evaluated, misidentified components are detected, and this feedback is automatically used to generate retraining data. The multi-anchor matching algorithm provides continuous feedback to improve the model's accuracy over time without user involvement.
2Ease of manufacture
If the CV model is trained on design-time user interfaces, then the model can be developed and tested beforehand, but the model becomes suboptimal when runtime interfaces differ from design-time interfaces
Solution Approach 1:
The system dynamically adapts the CV model by continuously capturing runtime interface screenshots and using them for retraining. The multi-anchor matching algorithm dynamically adjusts to interface changes by learning from actual runtime variations, making the model flexible and adaptable rather than static and rigid.
Solution Approach 2:
The system performs preliminary actions by maintaining a repository of interface variations captured at runtime before they cause failures. These pre-captured screenshots and identified components are ready for immediate retraining, allowing the model to proactively adapt to anticipated interface changes rather than reactively failing.
3Productivity
If synthetic data is used to train the CV model, then training data can be generated without collecting actual screenshots, but the model may not generalize well to real-world variations
Solution Approach 1:
The system creates accurate copies of real runtime interface screenshots for training purposes. Instead of using synthetic generated images, the system captures actual screenshots from the application environment, ensuring that training data faithfully represents real-world variations while maintaining high identification accuracy.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A Computer Vision (CV) model generated by a Machine Learning (ML) system may be retrained for more accurate computer image analysis in Robotic Process Automation (RPA). A designer application may receive a selection of a misidentified or non-identified graphical component in an image form a user, determine representative data of an area of the image that includes the selection, and transmit the representative data and the image to an image database. A reviewer may execute the CV model, or cause the CV model to be executed, to confirm that the error exists, and if so, send the image and a correct label to an ML system for retraining. While the CV model is being retrained, an alternative image recognition model may be used to identify the misidentified or non-identified graphical component.