Speech Processing with Widget Context for Accurate Skill Routing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech processing systems struggle to seamlessly integrate with multi-modal widgets on devices, limiting user multitasking and efficient interaction switching between skills and widgets.
Innovation Solution
A system that utilizes widget context data to interpret natural language inputs, determining intents and entities based on the content displayed in multi-modal widgets, and selects appropriate skills for response, allowing users to interact with multiple widgets and skills efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If speech processing systems use traditional skill-based interaction models, then they can maintain simple system architecture, but they fail to integrate seamlessly with multi-modal widgets and limit user multitasking capability
Solution Approach 1:
The speech processing system is enhanced to handle multiple interaction modes simultaneously - traditional skill-based voice commands and widget-based visual interactions. The system universally processes both audio inputs and visual context from multi-modal widgets, enabling seamless integration without requiring separate processing paths for different interaction types
Solution Approach 2:
Widget context data serves as an intermediary layer between the speech processing system and multi-modal widgets. This context data structure mediates the interaction by carrying relevant information from widgets to the speech processing engine, enabling intelligent routing and response generation without direct complex system integration
2Measurement precision
If the system processes speech inputs without considering widget context, then processing speed is faster, but the ability to interpret user intent accurately deteriorates
Solution Approach 1:
Widget context data is prepared and made available in advance before speech processing occurs. The system pre-loads and structures relevant widget information so that when speech input arrives, the processing engine can immediately access contextual information without performing time-consuming queries or data gathering during the critical processing window
Solution Approach 2:
The system selectively retrieves only the specific widget context data relevant to the current speech input rather than processing all available widget information. This localized approach to context retrieval maintains high intent interpretation accuracy by focusing on relevant data while minimizing processing overhead from unnecessary context analysis
3Productivity
If users can only interact with one skill at a time, then the system maintains simple interaction logic, but user multitasking efficiency is reduced
Solution Approach 1:
The system dynamically determines which widget or skill should receive the speech input based on real-time analysis of widget context data and current user interactions. This dynamic routing capability enables seamless switching between multiple widgets and skills without requiring complex pre-configured interaction paths or rigid single-task constraints
Solution Approach 2:
Multiple widget interactions and skill-based commands are merged into a unified processing framework. The system combines widget context data with traditional speech recognition outputs, allowing users to interact with multiple widgets simultaneously through natural speech while maintaining coherent interaction state across different modalities
Data Source
AI summary
Techniques for performing speech processing using multi-modal widget information are described. A system may receive input data corresponding to a user input. The system may also receive widget context data corresponding to one or more multi-modal widgets active at a device. The system may use the widget context data to perform natural language understanding (NLU) processing with respect to the user input, and for selecting a skill component for responding to the user input. The system may send a widget identifier to the skill component when invoking the skill to respond to the user input.