Vector Multiply Unit for CNN Matrix Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Convolutional neural networks (CNNs) face significant computational and data complexity due to their wide and deep network structures, leading to performance bottlenecks and high computational requirements, especially in matrix multiplication operations, which existing hardware implementations, such as FPGAs and ASICs, struggle to efficiently address.
Innovation Solution
An integrated circuit with a vector multiply unit featuring a plurality of multiply/accumulate nodes that performs outer products of input vectors, along with data feeders that provide data in vector format, optimizing the processing of convolution and fully connected layers by efficiently interleaving weights and data for faster matrix operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional FPGA or ASIC implementations are used for CNN matrix operations, then the system provides dedicated hardware acceleration, but the performance is limited to 10's or 100's GOPS and severely bottlenecked by data movements
Solution Approach 1:
The patent segments the input data and weight matrices into multiple tiles that can be processed in parallel. The matrix multiplication is divided into block-wise operations where sub-matrices are processed independently and concurrently, allowing the system to overcome data movement bottlenecks by distributing computation across multiple processing elements.
Solution Approach 2:
The patent introduces a spatial dimension to data processing by organizing computations in a two-dimensional mesh topology of processing elements. Data flows through multiple dimensions (row-wise and column-wise) enabling parallel processing across the network, transforming a sequential data movement problem into a parallel spatial computation problem.
2Reliability
If wide and deep CNN networks are used, then robust feature identification is achieved, but computational complexity reaches hundreds of GOPS or TOPS
Solution Approach 1:
The patent applies segmentation by dividing large matrix operations into smaller tile-based computations. This allows deep and wide networks to be processed in manageable blocks, reducing the peak computational power requirement while maintaining the overall accuracy of feature identification through systematic processing of all network layers.
Solution Approach 2:
The patent performs preliminary tiling and organization of weight and input data before computation. By pre-processing and arranging data in optimal formats for parallel processing, the system reduces the computational burden during actual inference, enabling deployment of wide and deep networks with reduced real-time power requirements.
3Measurement precision
If fully connected layers are implemented with huge data bandwidth, then classification accuracy is improved, but data movement requirements increase significantly
Solution Approach 1:
The patent segments the large-scale matrix operations in fully connected layers into tile-based computations. This segmentation reduces the instantaneous data bandwidth requirement by processing data in smaller, manageable blocks that can be handled by available memory interfaces, while still achieving the same classification accuracy through complete processing of all input features.
Data Source
AI summary
Described examples include an integrated circuit including a vector multiply unit including a plurality of multiply/accumulate nodes, in which the vector multiply unit is operable to provide an output from the multiply/accumulate nodes, a first data feeder operable to provide first data to the vector multiply unit in vector format, and a second data feeder operable to provide second data to the vector multiply unit in vector format.