Human-in-the-Loop Voice Automation for Cross-App Actions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing mobile application automation systems lack the ability to efficiently automate user interface actions across multiple applications and require extensive manual intervention, especially when dealing with complex user journeys and private data handling.
Innovation Solution
A computing device with a system-level assistant component and an application-level assistant component, connected by a trusted intermediary component, enables automation of user interface actions on mobile applications, allowing for human-in-the-loop functionality to manage complex tasks and ensure secure data handling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If a system-level assistant component is introduced to enable cross-application automation, then automation capability is improved, but device complexity increases
Solution Approach 1:
The system is divided into distinct components: application-level assistant components within individual applications and a system-level assistant component at the OS level. This segmentation allows each component to have specialized functionality while working together through standardized interfaces, enabling cross-application automation without creating a monolithic complex system.
Solution Approach 2:
The system-level assistant component acts as an intermediary between multiple application-level assistant components and the operating system. It receives automation requests from one application, processes them through the trusted intermediary, and executes actions in other applications, thereby enabling cross-application automation while maintaining system architecture integrity.
2Productivity
If automated processes are implemented across multiple applications, then productivity is improved, but security risks increase due to private data handling
Solution Approach 1:
A trusted intermediary component is introduced between the system-level assistant and application-level assistants to handle data transmission. This intermediary validates and secures the exchange of information, ensuring that private data is protected during automated cross-application processes while maintaining automation efficiency.
Solution Approach 2:
Different security measures are applied to different data types and applications involved in the automation process. The system implements context-aware security protocols that adapt to the specific privacy requirements of each application and data type, allowing automated processes to proceed with appropriate security controls for each local context.
3Ease of operation
If application-level assistant components are used within individual applications, then ease of operation is improved, but automation scope is limited to single applications
Solution Approach 1:
The system-level assistant component provides universal functionality that enables application-level assistants to work together across multiple applications. It implements a standardized interface and communication protocol that allows automated tasks to span across different applications, making the automation system versatile while maintaining the ease of operation of individual application interfaces.
Data Source
AI summary
The present disclosure is directed to automating user interactions in an application. In particular, the computing device can determine, based on the input data, an intent associated with performing an action on the computing device. Additionally, the computing device can invoke, using an application-level assistant component configured to provide an application-level interface on the computing device, execution of an application. Moreover, the computing device can transmit, to a system-level assistant component configured to provide a system-level interface with the computing device, instructions to interface with an actionable component of the application. The instructions can be transmitted using a trusted intermediary component configured to facilitate an interface between the system-level assistant component and the application-level assistant component. Furthermore, the computing device can engage, using the system-level assistant component, the actionable component of the application to perform the action.