Automatic Speech Recognition Acceleration via Key Frame Extraction

Solution Overview

Problem

Current automatic speech recognition (ASR) systems based on encoder-decoder architectures with attention mechanisms require significant computational resources and time to process audio corpora, necessitating an efficient acceleration method.

Innovation Solution

The proposed method involves preprocessing acoustic signals by padding and compressing them to reduce size, processing the compressed sequences through a pre-trained encoder neural network, decompressing the outputs to restore original ordering, and then inputting them into a decoding module for recognition text generation, utilizing techniques like zero-frame removal and restoration and offset mask matrices to optimize computation and memory usage.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If encoder-decoder architecture with attention-mechanism is used for ASR, then recognition accuracy is improved, but processing time and computation resources increase significantly

Engineering Contradiction:

Improverecognition accuracyVSAvoidprocessing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The audio sequence is divided into multiple frames, with selected key frames extracted and processed. Instead of processing the entire audio sequence uniformly, the method segments the data temporally by identifying and selecting important time points (key frames) that capture essential speech information, thereby reducing the overall processing burden while maintaining recognition accuracy.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Key frames are extracted from the complete audio sequence based on importance criteria. The method identifies and extracts only the most relevant frames containing critical speech information, discarding redundant data. This extraction process reduces the input size for the encoder-decoder model while preserving the essential features needed for accurate speech recognition.

Inventive Principle:

Principle #2Taking out (Extraction)

2Measurement precision

If encoder-decoder architecture with attention-mechanism is used for ASR, then recognition accuracy is improved, but computation resources increase significantly

Engineering Contradiction:

Improverecognition accuracyVSAvoidcomputation resources

Core Design Contradiction:

Measurement precisionVSUse of energy by moving object

Solution Approach 1:

The audio sequence is divided into multiple frames, with selected key frames extracted and processed. Instead of processing the entire audio sequence uniformly, the method segments the data temporally by identifying and selecting important time points (key frames) that capture essential speech information, thereby reducing the overall processing burden while maintaining recognition accuracy.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Key frames are extracted from the complete audio sequence based on importance criteria. The method identifies and extracts only the most relevant frames containing critical speech information, discarding redundant data. This extraction process reduces the input size for the encoder-decoder model while preserving the essential features needed for accurate speech recognition.

Inventive Principle:

Principle #2Taking out (Extraction)

3Reliability

If padded sequence is processed directly through encoder neural network, then all audio data is analyzed, but execution time and memory consumption increase

Engineering Contradiction:

Improvecompleteness of analysisVSAvoidexecution time

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

Key frames are extracted from the complete audio sequence based on importance criteria. The method identifies and extracts only the most relevant frames containing critical speech information, discarding redundant data. This extraction process reduces the input size for the encoder-decoder model while preserving the essential features needed for accurate speech recognition.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

Different parts of the audio sequence are treated differently based on their importance. Instead of uniform processing, the method applies selective attention by identifying key frames that require detailed analysis while skipping or summarizing less important segments. This local quality approach ensures critical speech information is fully analyzed while reducing overall processing time.

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS11830480B2Systems and methods for accelerating automatic speech recognition based on compression and decompression

Publication Date: 2023.11.28 BEIJING TRANSTREAMS TECH CO LTD

AI summary

Systems and methods are provided for automatic speech recognition. In the method, the system obtains a padded sequence by processing a plurality of acoustic signals. The system compresses the padded sequence by reducing the size of the padded sequence to obtain a compressed sequence. The system inputs the compressed sequence into a pre-trained encoder neural network to obtain an encoded sequence and then decompresses the encoded sequence by recovering the encoded sequence to an original sequential ordering. The system inputs the encoded sequence to a decoding module to obtain recognition texts.