Matrix Processor Swizzled Weights for Neural Network Convolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning technologies face challenges in providing flexible and efficient hardware for neural network convolution, particularly due to the 'halo problem' and the need to handle various types of computational layers with non-standard dimensions, which current solutions like GPUs and custom accelerators fail to address effectively.
Innovation Solution
A matrix-processor-based device is configured with a CPU and matrix processor that reorganizes weight and input matrices into swizzled matrices with regular dimensions, allowing for efficient matrix multiplication and accumulation operations while handling irregular dimensions through nested loops, enabling flexible processing of convolution operations across different neural network layers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If GPUs are used to perform convolution through im2col preprocessing, then matrix multiplication can be performed efficiently, but memory and bandwidth are wasted due to data duplication
Solution Approach 1:
The patent segments the convolution operation into two distinct phases: a preprocessing phase that converts the input image into a format suitable for the matrix processor, and a matrix multiplication phase that performs the actual convolution computation. This segmentation allows the system to avoid duplicating entire weight matrices while still achieving efficient convolution processing through specialized preprocessing of the input data.
Solution Approach 2:
The patent applies preliminary action by performing the im2col transformation before the matrix multiplication phase. The input image is preprocessed to convert it into a format that can be directly multiplied with the weight matrix, eliminating the need for data duplication during the main computation phase and reducing memory bandwidth requirements.
2Productivity
If custom convolution hardware accelerators are designed, then convolution processing can be optimized, but flexibility to handle different convolution types and non-standard dimensions is limited
Solution Approach 1:
The patent implements universality by designing a matrix processor that can handle multiple types of convolution operations (2D convolution, 1D convolution, fully connected layers, recurrent neural network layers) using the same hardware architecture. The processor uses a unified matrix multiplication core with configurable preprocessing to accommodate different convolution types and non-standard dimensions, eliminating the need for separate hardware accelerators for each operation type.
Solution Approach 2:
The patent applies dynamics by making the preprocessing stage configurable and adaptable. The system can dynamically adjust the preprocessing parameters and matrix dimensions based on the specific convolution operation being performed, allowing the same hardware to efficiently handle various convolution types and non-standard layer dimensions without requiring custom hardware designs for each case.
3Productivity
If matrices are split into smaller submatrices to handle large dimensions, then processing can be performed, but the halo problem occurs requiring data sharing among matrix processors
Solution Approach 1:
The patent applies preliminary action by performing the im2col transformation before dividing the matrix into submatrices. This preprocessing step reorganizes the input data in advance, which simplifies the subsequent matrix multiplication and reduces the complexity of data sharing requirements among matrix processors handling submatrices.
Data Source
AI summary
Providing flexible matrix processors for performing neural network convolution in matrix-processor-based devices is disclosed. In this regard, a matrix-processor-based device provides a central processing unit (CPU) and a matrix processor. The matrix processor reorganizes a plurality of weight matrices and a plurality of input matrices into swizzled weight matrices and swizzled input matrices, respectively, that have regular dimensions natively supported by the matrix processor. The matrix-processor-based device then performs a convolution operation using the matrix processor to perform matrix multiplication/accumulation operations for the regular dimensions of the weight matrices and the input matrices, and further uses the CPU to execute instructions for handling the irregular dimensions of the weight matrices and the input matrices (e.g., by executing a series of nested loops, as a non-limiting example). The matrix-processor-based device thus provides efficient hardware acceleration by taking advantage of dimensional regularity, while maintaining the flexibility to handle different variations of convolution.