Mobile Assistant Action Recording for Reliable Voice Talkback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional digital assistants face issues with privacy concerns, misinterpretations of spoken commands, unavailability due to weak signals, and the requirement for structured dialects, limiting their functionality to predefined actions and lack of verbal feedback on mobile devices.
Innovation Solution
A crowd-sourced digital assistant that allows users to create and distribute actions via verbal commands, leveraging machine learning to customize and optimize commands based on installed applications, reducing redundancy and enhancing privacy by utilizing existing device applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional digital assistants use predefined actions and structured dialects, then command interpretation reliability is improved, but adaptability to user needs deteriorates
Solution Approach 1:
The digital assistant dynamically adapts its command structure and action set based on user interactions and device context. The system learns from user behavior patterns and automatically adjusts the dialect requirements and available actions, transitioning from a static predefined structure to a dynamic, user-specific configuration that maintains reliability while improving adaptability
Solution Approach 2:
The system changes parameters such as command structure flexibility, action availability, and response format based on user profile and device state. By adjusting these parameters dynamically, the assistant maintains reliable command interpretation while adapting to diverse user needs and preferences
2Adaptability or versatility
If digital assistants provide comprehensive actions across multiple applications, then versatility is improved, but device complexity increases
Solution Approach 1:
The digital assistant achieves comprehensive action coverage by universally integrating with existing applications on the device rather than maintaining separate action modules. The system leverages the multi-functionality of standard application interfaces to provide broad capabilities without duplicating functionality, thereby reducing overall system complexity
Solution Approach 2:
The assistant acts as an intermediary layer that translates user commands into application-specific actions through standardized interfaces. This mediator approach allows comprehensive action coverage across multiple applications while abstracting the complexity of individual application implementations from the user and the core assistant system
3Speed
If digital assistants execute actions directly on mobile devices, then response speed is improved, but privacy concerns increase
Solution Approach 1:
The digital assistant performs self-service by executing actions directly on the user's device using locally available applications and data. This eliminates the need to transmit sensitive information to remote servers, maintaining fast response speeds while addressing privacy concerns through local-only processing of personal information
Data Source
AI summary
Embodiments of the present invention provide systems, methods, and computer storage media directed to providing talk back automation for applications installed on a mobile device. To do so actions (e.g., talk back features) can be created, via the digital assistant, by recording a series of events that are typically provided by a user of the mobile device when manually invoking the desired action. At a desired state, the user may select an object that represents the output of the application. The recording embodies the action and can be associated with a series of verbal commands that the user would typically announce to the digital assistant when an invocation of the action is desired. In response, the object is verbally communicated to the user via the digital assistant, a different digital assistant, or even another device. Alternatively, the object may be communicated to the same application or another application as input.