Voice Assistant Control of GUI Apps Without API Integration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automated assistants have limited functionality when interacting with separate applications, often requiring an application programming interface (API) that many applications lack, leading to inefficiencies and resource waste due to interrupted interactions and unfulfilled user commands.
Innovation Solution
Utilizing trained machine learning models to map natural language inputs to graphical user interface elements, enabling automated assistants to control separate applications without requiring an API, by correlating spoken commands to actionable GUI elements within an embedding space.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If automated assistants use traditional API-based control methods, then they can reliably control applications with automated assistant API functionality, but they cannot control separate applications that lack such API functionality
Solution Approach 1:
The patent introduces an intermediary system that bridges the automated assistant and separate applications without requiring direct API integration. This intermediary captures user interactions with the separate application, trains machine learning models to replicate the application's control logic, and enables the automated assistant to control the application through learned patterns rather than formal APIs, thus resolving the contradiction between versatility and reliability
Solution Approach 2:
The patent replaces the traditional mechanical/API-based control system with a data-driven machine learning system. Instead of relying on predefined API contracts, the system uses trained models to predict and execute control actions based on natural language inputs and observed user behavior patterns, enabling control of applications that lack formal automation interfaces
2Ease of operation
If automated assistants interrupt application operations to handle user commands, then they can respond to user requests, but they waste computational resources and cause data loss when users actually intend to perform application tasks
Solution Approach 1:
The patent implements preliminary action by training machine learning models in advance using captured user interaction data with the separate application. This pre-training enables the automated assistant to understand application control patterns before actual use, allowing it to distinguish between genuine assistant requests and application tasks, thereby reducing unnecessary interruptions and improving resource efficiency
Solution Approach 2:
The system incorporates feedback mechanisms where user interactions with the separate application are captured and used to continuously improve the machine learning models. This feedback loop enables the automated assistant to learn from actual user behavior patterns, becoming more accurate in determining when to interrupt application operations and when to allow users to complete their tasks independently
3Adaptability or versatility
If automated assistants require applications to have automated assistant API functionality, then they ensure standardized control interfaces, but they limit the assistant's functionality to only those applications with such APIs
Solution Approach 1:
The patent implements universality by creating a single machine learning-based control framework that can work with any separate application regardless of whether it has native automated assistant API support. The system captures interactions across different application types and trains universal models that can generalize control strategies, enabling one automated assistant instance to control multiple diverse applications without requiring application-specific API integrations
Solution Approach 2:
The system uses copying by replicating user interaction patterns and application control logic through machine learning models. Instead of requiring each application to expose its control interface via API, the system observes and copies user behavior patterns, then uses these copied patterns to control the application, effectively creating a virtual API layer that mirrors the application's actual control mechanisms
Data Source
Figure 1A
Figure 1B
Figure 1C
AI summary
Implementations relate to an automated assistant that is capable of interacting with non-assistant applications that do not have functionality explicitly provided for interfacing with certain automated assistants. Application data, such as annotation data and/or GUI data, associated with a non-assistant application, can be processed to map such data into an embedding space. An assistant input command can then be processed and mapped to the same embedding space, and a distance from the assistant input command embedding and the non-assistant application data embedding can be determined. When the distance between the assistant input command embedding and the non-assistant application data embedding satisfies threshold(s), the automated assistant can generate instruction(s), for the non-assistant application, that correspond to the non-assistant application data. For instance, the instruction(s) can simulate user input(s) that cause the non-assistant application to perform one or more operations characterized by, or otherwise associated with, the non-assistant application data.