Hybrid GPU CPU Graph Traversal for Speech Recognition Memory Limits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for large-scale graph traversal on parallel processor platforms, such as those used in large vocabulary continuous speech recognition, are limited by memory constraints, particularly in manycore GPU architectures, which struggle with extremely large language models and acoustic models that exceed the available memory capacity.
Innovation Solution
The implementation of an on-the-fly hypothesis rescoring method that leverages both GPUs and CPUs, where a small WFST search graph is composed for GPU processing, and a larger language model is used for rescoring, utilizing model likelihood corrections to handle large models efficiently, and employing data structures like HASH-MAP, STATIC ARRAY, and BINARY-SEARCH TREE to optimize memory usage and processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single fully composed WFST search graph is used for speech recognition, then decoding efficiency is improved, but memory requirements become infeasibly large for large models
Solution Approach 1:
The patent divides the single large WFST search graph into multiple smaller sub-WFST search graphs, each representing a subset of the vocabulary or model components. This segmentation allows the system to process large-scale speech recognition tasks by breaking down the infeasibly large monolithic graph into manageable pieces that can fit in GPU memory while maintaining the overall recognition capability through coordinated processing of multiple segments.
2Quantity of substance
If multiple sub-WFSTs are composed on-the-fly during decoding, then memory usage is reduced, but decoding speed becomes significantly slower
Solution Approach 1:
The patent pre-composes multiple sub-WFST search graphs during an offline preparation phase, organizing them into a structured collection that can be efficiently accessed during decoding. This preliminary action eliminates the need for time-consuming on-the-fly composition during real-time decoding, thus maintaining high decoding speed while keeping memory usage low by only loading necessary sub-graphs.
Solution Approach 2:
The system dynamically selects and activates specific sub-WFST search graphs based on the current decoding context and vocabulary subset being processed. This dynamic approach allows the decoder to efficiently navigate through different portions of the large vocabulary space by switching between pre-composed sub-graphs, maintaining both memory efficiency and decoding speed.
3Quantity of substance
If GPU memory is increased to accommodate large language models, then model capacity is improved, but cost and accessibility become limiting factors
Solution Approach 1:
By segmenting the large language model into multiple smaller sub-WFSTs, the patent enables the system to run large-scale models on standard GPU hardware with limited memory (2-8 GB). This segmentation strategy allows researchers and practitioners to utilize readily available GPU resources rather than requiring expensive, specialized high-memory hardware, thus improving accessibility and reducing costs.
Data Source
Figure 1
Figure 2~3
Figure 4~5
AI summary
The present invention describes methods for performing large-scale graph traversal calculations on parallel processor platforms. The invention describes methods for on-the-fly hypothesis rescoring that utilizes graphic processing units (GPUs) in combination with utilizing central processing units (CPUs) of computing devices. The invention is described in one embodiment as applied to the task of large vocabulary continuous speech recognition.