Vision and Language-Assisted Smartphone Task Automation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current smartphone automation technologies are limited in their ability to automate tasks across various third-party applications due to their inability to recognize and adapt to different user inputs, UI elements, positional changes, and visual changes, and struggle to understand user utterances for new automation scripts.
Innovation Solution
A vision and language-assisted smartphone task automation system that uses a Programming-by-Demonstration approach with a combination of natural language understanding and object detection neural networks to create and execute automation scripts for arbitrary tasks across multiple third-party applications, allowing users to create scripts through user interactions and adapt to changes in UI elements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional automation scripts are used to control third-party applications, then automation capability is limited to built-in applications and a small number of integrated web services, but the system cannot adapt to significant variations in applications and tasks
Solution Approach 1:
The patent introduces an intermediary system consisting of a screen recorder, neural network model, and script generator that mediates between user actions and automation execution. This intermediary layer captures screen content, processes it through neural networks to understand UI elements and user intent, and generates adaptive scripts without requiring direct integration with each third-party application, thus achieving versatility while managing complexity.
Solution Approach 2:
The patent creates a universal automation system that can handle multiple third-party applications through a single integrated platform. The neural network model is trained to recognize various UI elements across different applications, and the script generator produces universal automation scripts that can adapt to different app contexts, eliminating the need for application-specific automation solutions.
2Reliability
If automation scripts are created for each specific task, then task automation is achieved, but the system cannot recognize changes in UI elements and positional changes
Solution Approach 1:
The patent implements dynamic automation scripts that can adapt to changing UI conditions. The neural network model continuously processes screen content during execution, identifies current UI element positions and characteristics, and adjusts the automation script accordingly. This dynamic adaptation allows the system to maintain reliability despite UI element repositioning or visual changes in third-party applications.
3Ease of operation
If users create automation scripts through demonstration, then ease of use is improved for non-expert users, but the system cannot understand user utterances for new automation scripts
Solution Approach 1:
The patent merges two approaches: programming-by-demonstration (capturing user actions via screen recording) and natural language processing (analyzing user utterances). The system combines visual information from screen content with linguistic information from user speech or text, using neural networks to integrate both modalities and accurately infer user intent, thereby maintaining ease of use while improving understanding capability.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
An apparatus for vision and language-assisted smartphone task automation, includes a processor configured to execute instructions to identify whether an utterance of a user refers to one among at least one existing automation script, and based on the utterance being identified to not refer to the one among the at least one existing automation script, capture one or more screenshots and one or more coordinates of a display, the one or more screenshots and the one or more coordinates corresponding to one or more interactions of the user with the display while the user is demonstrating a new automation script, perform object detection and optical character recognition on the captured one or more screenshots and the captured one or more coordinates to obtain user interface (UI) elements corresponding to the one or more interactions, and obtain the new automation script, based on the obtained UI elements.