Semantic Similarity Caching for NLU Inference Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech processing systems face high computational costs due to the execution of numerous natural language understanding (NLU) processes for each user utterance, even though only a small number of skills are typically applicable.
Innovation Solution
A cache-based skill selector is implemented to store historical skills used for processing specific natural language inputs. When a similar utterance is detected, the system retrieves the list of previously used skills from the cache, reducing the need to execute all NLU processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If all NLU processes are executed for each user utterance to ensure comprehensive skill coverage, then the system can handle diverse commands, but the computational cost and processing time increase significantly
Solution Approach 1:
The system performs preliminary action by caching the set of applicable skills for each utterance template in advance. When a user speaks, the system retrieves the pre-computed skill set from the cache rather than executing all NLU processes, thus preparing the solution beforehand to avoid redundant computation.
Solution Approach 2:
The system applies local quality by providing different processing paths based on the specific utterance type. For cached utterances, only the relevant cached skill set is processed, while for uncached utterances, full NLU processing is performed. This localized optimization ensures comprehensive coverage where needed while improving efficiency for common cases.
2Measurement precision
If multiple NLU processes are executed to accurately determine the appropriate skill, then the semantic interpretation is more precise, but the latency increases
Solution Approach 1:
The system pre-computes and caches the set of applicable skills for each utterance template during system initialization or updates. This preliminary action stores the results of complex NLU analysis in advance, allowing rapid retrieval during actual user interactions without re-executing the full analysis.
Solution Approach 2:
The system creates and stores copies of skill sets associated with each utterance template in a cache. These cached copies are retrieved and used for matching during runtime, avoiding the need to re-execute the original NLU processes that generated them, thus maintaining accuracy while reducing latency.
3Productivity
If the system caches skill sets for each utterance template to reduce computation, then processing speed improves, but the memory usage increases
Solution Approach 1:
The system performs preliminary computation of skill sets and caches them before runtime processing. By pre-computing these data structures during system initialization or updates, the expensive NLU analysis is avoided during user interactions, trading upfront computation and memory allocation for faster runtime performance.
Data Source
AI summary
Devices and techniques are generally described for inference reduction in natural language processing using semantic similarity-based caching. In various examples, first automatic speech recognition (ASR) data representing a first natural language input may be determined. A cache may be searched using the first ASR data. A first skill associated with the first ASR data may be determined from the cache. In some examples, first intent data representing a semantic interpretation of the first natural language input data may be determined by using a first natural language process associated with the first skill.