Parallel LSTM Processing Unit with On-Chip Buffering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for performing LSTM neural network operations, such as using general-purpose processors or GPUs, face performance bottlenecks due to low operation efficiency, high power consumption, and limited on-chip buffer capacity, leading to inefficient parallelism and increased IO overhead.
Innovation Solution
A device and method featuring parallel data buffer units and data processing modules that perform LSTM operations using vector point-multiplication, vector addition, vector summation, and vector nonlinear function transformation, with weights and biases stored in data buffer units to reduce IO operations and enable efficient parallel processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If general-purpose processors are used to implement LSTM network operations, then the system has high flexibility and can support various algorithms, but the operation performance is relatively low and cannot be accelerated by normal parallelism
Solution Approach 1:
The patent replaces the general-purpose processor's sequential instruction execution mechanism with a dedicated parallel computing architecture that uses multiple processing elements operating simultaneously on multiple data elements, substituting the mechanical instruction decoding and execution pipeline with hardware-accelerated parallel computation units optimized for neural network operations
Solution Approach 2:
The patent segments the LSTM computation into independent parallel operations by dividing the data into multiple chunks that can be processed simultaneously by different processing elements, each handling specific matrix multiplications and activation functions in parallel, thereby achieving acceleration through normal parallelism
2Productivity
If multiple general-purpose processors are used in parallel to improve performance, then the operation performance increases, but mutual communication among processors becomes a performance bottleneck
Solution Approach 1:
The patent merges multiple processing elements into a unified parallel processing architecture with shared memory access, combining the computational units and their data paths into an integrated system that eliminates the need for complex inter-processor communication protocols and reduces communication overhead through direct memory access and shared resources
3Adaptability or versatility
If general-purpose processors decode neural network operations into instructions, then the system can execute diverse operations, but the front-end decoding has large power consumption overhead
Solution Approach 1:
The patent substitutes the software-based instruction decoding mechanism with hardware-based direct execution, where the control unit directly interprets high-level neural network operations and generates control signals for the processing elements without requiring multi-stage instruction decoding, thereby significantly reducing power consumption in the front-end
4Productivity
If GPU is used to perform LSTM operations with generic SIMD instructions, then the system can execute parallel operations, but a large amount of front-end decoding is still required which brings extra overhead
Solution Approach 1:
The patent replaces the GPU's generic SIMD instruction decoding mechanism with a dedicated control unit that directly generates control signals for parallel processing elements based on simplified operation codes, substituting the complex multi-stage decoding pipeline with a streamlined hardware control path that reduces overhead while maintaining parallel execution capability
Solution Approach 2:
The patent optimizes the control mechanism locally by designing a specialized control unit tailored specifically for LSTM operations, with hardwired logic for generating control signals for matrix multiplications, activation functions, and gate operations, rather than using a general-purpose instruction decoder that must handle all possible operations
5Area of stationary object
If GPU has small on-chip buffer, then the chip area is reduced, but relevant parameters need to be repeatedly transferred from outside the chip making off-chip bandwidth a performance bottleneck
Solution Approach 1:
The patent implements preliminary action by pre-loading all necessary weights, biases, and input data into the on-chip buffer before computation begins, and by organizing the data in a format optimized for the parallel processing elements, thereby eliminating the need for repeated data transfers during the computation and maximizing the utilization of the limited on-chip memory
Data Source
Figure 1~2
Figure 3
Figure 3
AI summary
A device and a method for performing LSTM neural network operations are provided. The device includes a direct memory access unit, an instruction buffer unit, a controller unit, a plurality of data buffer units arranged in parallel, and a plurality of data processing modules arranged in parallel, wherein the plurality of data processing modules have one-to-one correspondence with the data buffer units, for acquiring input data as well as weights and biases required in operations from corresponding data buffer units, and performing the LSTM neural network operations; parallel operations are performed among the plurality of data processing modules. The present disclosure operates with dedicated instructions, which greatly reduces the number of instructions required for operations and reduces the decoding overhead; weights and biases are buffered, such that the overhead in data transmission is reduced; the present disclosure is not limited to specific application fields, and can be used in such fields as speech recognition, text translation, and music synthesis for its strong scalability; the plurality of data processing modules operate in parallel, which significantly improves calculation speed of the LSTM network.