Pre-caching Speech Feature Data to Reduce Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Automatic speech recognition (ASR) systems experience latency due to the need to retrieve and process Natural Language Understanding (NLU) data at the time of ranking, which introduces delays in speech processing systems.
Innovation Solution
Implementing a pre-fetching and caching system to pre-compute and cache feature data before it is used by downstream speech processing components, allowing for local and distributed caching to reduce latency and improve responsiveness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If feature data is computed and retrieved at the time of ranking, then the speech processing system can maintain simplicity in architecture, but the system experiences increased latency and reduced responsiveness
Solution Approach 1:
The system pre-computes feature data from source data before it is actually needed for ranking. Upstream speech processing components generate and cache feature data in advance, so that when downstream components need this data for request processing, it is already available, thereby reducing latency without significantly increasing system complexity
Solution Approach 2:
The patent introduces a caching mechanism as an intermediary layer between upstream feature generation components and downstream ranking components. This cache stores pre-computed feature data and facilitates efficient retrieval, acting as a mediator that decouples the timing of feature computation from feature usage, thus reducing the critical path latency
2Productivity
If pre-fetching and caching of feature data is implemented, then speech processing latency is reduced, but the system complexity and infrastructure requirements increase
Solution Approach 1:
The caching system is segmented into multiple independent caches distributed across different speech processing components. Each component maintains its own cache of pre-computed features, allowing parallel operation and reducing the complexity burden on any single component while collectively improving system throughput
Solution Approach 2:
Each speech processing component autonomously manages its own feature cache, pre-computing and storing features that it anticipates will be needed. This self-service approach eliminates the need for a centralized cache management system, reducing overall infrastructure complexity while still achieving the throughput benefits of pre-computation
Data Source
AI summary
Devices and techniques are generally described for pre-caching of speech processing feature data. In various examples, first data indicating source data is received from a first speech processing component. The source data may be used to generate first feature data. In various examples, a first request to process first input data is received. A second speech processing component may generate the source data during processing of the first input data. The first feature data may be generated using the source data. The first feature data may be sent to the first speech processing component. In some examples, the first speech processing component may store the first feature data in a first cache local to the first speech processing component.