Mobile App Talkback Actions for Voice Command Automation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional digital assistants face limitations in executing actions within mobile applications, suffer from privacy concerns, and require users to structure their commands in a specific dialect, leading to misinterpretations and frustration.
Innovation Solution
A crowd-sourced digital assistant that allows users to create and distribute actions through verbal commands, leveraging machine learning to customize and optimize commands based on user preferences and application usage, reducing redundancy and enhancing privacy by utilizing existing applications on user devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If users manually review and respond to each action notification, then they can provide informed feedback, but it consumes significant time and effort
Solution Approach 1:
The system enables automated self-service by allowing actions to automatically respond to their own notifications based on pre-configured rules and machine learning models, eliminating the need for manual user intervention while maintaining reliable feedback loops
Solution Approach 2:
The system implements automated feedback mechanisms where action notifications are automatically responded to with appropriate actions based on analyzed context, user preferences, and learned patterns, creating efficient closed-loop feedback without manual time investment
2Loss of information
If the system sends detailed action notifications to keep users informed, then transparency is improved, but notification fatigue and user disengagement increase
Solution Approach 1:
The system applies partial action by selectively notifying users only about actions that require their attention or interest, filtering out routine automated responses, thereby maintaining information transparency for important matters while avoiding notification fatigue from mundane updates
Solution Approach 2:
The notification system segments information by priority, type, and user relevance, delivering customized notification subsets to different users based on their preferences and the significance of actions, ensuring critical information reaches users without overwhelming them
3Productivity
If the system automates action responses using machine learning, then time efficiency improves, but computational resources and model complexity increase
Solution Approach 1:
The system performs preliminary action by pre-training machine learning models offline and pre-configuring response rules before deployment, enabling efficient automated responses during operation without requiring complex real-time computation, thus balancing productivity with manageable system complexity
Data Source
Figure 1
Figure 2~3
Figure 4
AI summary
Embodiments of the present invention provide systems, methods, and computer storage media directed to providing talk back automation for applications installed on a mobile device. To do so actions (e.g., talk back features) can be created, via the digital assistant, by recording a series of events that are typically provided by a user of the mobile device when manually invoking the desired action. At a desired state, the user may select an object that represents the output of the application. The recording embodies the action and can be associated with a series of verbal commands that the user would typically announce to the digital assistant when an invocation of the action is desired. In response, the object is verbally communicated to the user via the digital assistant, a different digital assistant, or even another device. Alternatively, the object may be communicated to the same application or another application as input.