Embedding-Based LLM Tuning for Voice Device Shortlisting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing voice interface devices struggle to accurately determine which device should perform an action in response to a voice command, especially when multiple devices are associated with a user account, leading to a multitude of possible actions that overwhelm the language model's capabilities.
Innovation Solution
An embedding-based approach is employed to generate utterance, device, and API embeddings, utilizing a translation model trained with contrastive learning to associate utterances with relevant devices, and a shortlister to provide a subset of device information to the language model for precise action determination.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all device information is provided to the language model for action determination, then the model can consider all possible actions, but the computational load increases and accuracy decreases due to overwhelming possibilities
Solution Approach 1:
The patent segments device information into two groups: frequently accessed devices are stored in a first data structure (hash map) for quick access, while less frequently accessed devices are stored in a second data structure (sorted list) for systematic processing. This segmentation allows the system to prioritize processing of relevant devices while still considering all possible devices, thereby reducing computational load without sacrificing action determination accuracy.
Solution Approach 2:
The system performs preliminary actions by generating device embeddings and organizing device information into optimized data structures before the language model needs to process them. The embeddings are pre-computed and stored, allowing the model to quickly retrieve and process only the most relevant device information during action determination, rather than processing all raw device data from scratch each time.
2Measurement precision
If embedding dimension is increased to capture more device characteristics, then association accuracy improves, but computational resources and processing time increase
Solution Approach 1:
The patent implements parameter changes by allowing the embedding dimension to be dynamically adjusted based on the specific application requirements and available computational resources. The system can select different embedding dimensions (e.g., 768, 1024, or 2048) to balance between association accuracy and computational efficiency, optimizing performance for different deployment scenarios without being locked into a fixed high-dimensional representation.
Data Source
AI summary
Systems and methods for embedding-based LLM tuning include generating a first embedding of received user input data and utilizing a translation model trained to associate user input with device names to generate a second embedding that differs at least in part from the first embedding. Reference embeddings corresponding to devices associated with user account data may be generated and a subset of the reference embeddings that satisfy a threshold similarity to the second embedding may be determined. A large language model (LLM) configured to determine a response to the user input data may utilize data for a subset of devices that correspond to the subset of the reference embeddings for performing speech processing.