RNN-Based UI Workflow Prediction for Faster Repetitive Tasks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Human efficiency in performing repetitive tasks involving specific chronological steps and interactions with a user interface (UI) reaches a limit, necessitating additional computer assistance for further improvement.
Innovation Solution
A method and system that utilize a recurrent neural network (RNN) model to predict the next interaction of a user in a workflow, enabling auto focal adjustment, auto cursor snap, or providing a UI prompt to facilitate efficient workflow completion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If users perform repetitive tasks manually, then task completion is achieved, but efficiency reaches a limit and time is wasted
Solution Approach 1:
The system performs preliminary actions by predicting the next UI element user will interact with and automatically moving the cursor or adjusting focus to that element before the user needs it. This preliminary positioning of the cursor eliminates the time users would otherwise spend manually navigating to the next interaction point, thereby improving workflow efficiency without adding complexity.
2Productivity
If computer assistance is added to improve efficiency, then productivity increases, but device complexity increases
Solution Approach 1:
The system serves itself by using the user's own interaction patterns as training data to build predictive models. The system learns from observing how the user naturally interacts with the UI and then uses this learned knowledge to automatically assist the user. This self-service approach allows the system to improve productivity without requiring external complex control mechanisms, as the system adapts to the user's existing workflow rather than imposing a complex external structure.
3Speed
If cursor movement and focus adjustment are automated, then interaction speed increases, but ease of operation decreases due to potential inaccuracies
Solution Approach 1:
The system incorporates feedback mechanisms where user corrections or overrides of automated cursor movements are captured and used to retrain and refine the predictive model. When the user manually adjusts the cursor to a different element than predicted, this feedback is fed back into the system to improve future predictions. This continuous feedback loop ensures that the automation becomes increasingly accurate over time, maintaining ease of operation while preserving interaction speed benefits.
Data Source
AI summary
In several approaches, a processor feeds an interaction by a user into a recurrent neural network (RNN) model. A processor identifies, using the RNN model trained to identify a specific workflow based on the interaction, an in-process workflow of the user. A processor predicts, using the RNN model, a next interaction of the user based on a next expected step in the in-process workflow. A processor performs at least one optimization function related to the next interaction of the user including at least one of autofocusing on a next user interface (UI) element associated with the next expected step in the in-process workflow, snapping a text cursor to the next UI element associated with the next expected step in the in-process workflow, and providing a UI prompt with a link to the next UI element associated with the next expected step in the in-process workflow.