Automated UI Navigation Agents Using OCR and Deep Q-Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for automated user interface navigation, such as those using raw pixel data or heuristic-based macros, are inefficient and brittle, requiring significant computational resources and maintenance efforts due to their reliance on complex if-then statements and raw pixel data processing.
Innovation Solution
A neural network-based framework that simplifies the input state space to UI elements extracted via OCR and icon detection, and reduces the action space to UI elements and global actions, using deep Q-learning and behavior cloning to train agents with reduced training samples and iterations, enabling efficient and robust navigation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If raw pixel data is used as input for automated agent navigation, then sufficient information about current state is provided, but significant computational resources are required and processing becomes challenging
Solution Approach 1:
The patent extracts only the necessary information from raw pixel data by identifying and focusing on UI elements rather than processing the entire pixel matrix. This extraction approach maintains navigation reliability while significantly reducing computational resource requirements by processing only relevant portions of the visual input.
Solution Approach 2:
The patent introduces an intermediary processing layer that translates raw pixel data into a simplified representation of UI elements. This intermediary step acts as a bridge between the complex pixel data and the navigation agent, reducing computational burden while preserving essential navigation information.
2Ease of operation
If heuristic-based macros with if-then statements are used for automation, then coding control is achieved, but the system becomes highly brittle and requires continuous maintenance
Solution Approach 1:
The patent replaces static if-then rules with a dynamic neural network model that can adapt to new situations. The neural network learns patterns from training data and can generalize to unexpected states, making the automation system robust and flexible rather than brittle and rigid.
Solution Approach 2:
The patent enables the system to learn and improve automatically through training on demonstration data. Rather than requiring manual updates to if-then rules for each new scenario, the neural network self-adjusts its behavior through learning, reducing maintenance requirements while improving reliability.
3Adaptability or versatility
If complex if-then statements are used for macro automation, then scenario coverage is attempted, but creation and maintenance become inefficient
Solution Approach 1:
The patent uses copying by training the neural network on demonstration data that captures various scenarios. Instead of manually creating if-then rules for each scenario, the system learns from copied examples, achieving broad scenario coverage while dramatically improving creation efficiency.
Solution Approach 2:
The patent changes the fundamental parameters of the automation approach by transitioning from discrete if-then rules to a continuous neural network model. This parameter change allows the system to handle diverse scenarios through learned patterns rather than exhaustive rule coverage, improving both versatility and efficiency.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Provided is a framework to reliably build agents capable of user interface (UI) navigation. For example, example implementations create UI navigation agents with the power of neural networks that learn from human demonstrations.