Neural Network Acceleration via Group-Sparse Matrix Transformation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems lack efficient hardware support for executing neural networks that are both quantized and pruned, particularly on commodity CPUs, due to the limitations of existing hardware instructions and the inability to optimally utilize modern vectorial instruction sets like Intel AVX512 VNNI.
Innovation Solution
The method involves creating a synergy between pruning and quantization by tailoring these processes to the parameters of specialized vectorial instructions, such as FMA or MAC instructions, and employing vectorial instructions on compressed data to exploit the benefits of both compression techniques, including the use of group-sparse matrices and saliency-based pruning to optimize execution on CPUs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing hardware instructions (such as VNNI) are used for neural network execution, then vectorized computation efficiency is improved, but support for sparse computations is limited
Solution Approach 1:
The patent segments the computation into dense and sparse portions, processing them through different instruction paths. The system divides the neural network computation to leverage vectorized instructions for dense regions while handling sparse regions through specialized routines, thus resolving the contradiction between vectorized efficiency and sparse computation support
Solution Approach 2:
The patent introduces an intermediary layer that translates between dense vectorized operations and sparse computation requirements. This intermediary mechanism enables the system to use efficient vectorized instructions while still supporting sparse neural network operations through conversion and adaptation layers
2Quantity of substance
If both pruning and quantization are applied to compress neural networks, then model size and computation are reduced, but existing hardware lacks efficient support for executing compressed models
Solution Approach 1:
The patent changes the parameters of the computation to match hardware capabilities. By transforming pruned and quantized operations into formats that align with existing vectorized instruction sets, the system enables efficient execution of compressed models on standard hardware without requiring specialized processors
3Speed
If vectoral instructions are used for matrix multiplication, then execution speed is improved, but optimal utilization requires adaptation to specific instruction set parameters
Solution Approach 1:
The patent performs preliminary adaptation of the neural network computation to match vectorized instruction parameters before execution. By pre-configuring the computation structure to align with hardware instruction sets, the system achieves optimal speed utilization without requiring complex runtime adaptations
Data Source
AI summary
A system and method of accelerating execution of a NN model, by at least one processor may include: receiving a first matrix A, representing elements of a kernel K of the NN model and a second matrix B, representing elements of an input I to kernel K; producing from matrix A, a group-sparse matrix A′, comprising G tensors of elements. The number of elements in each tensor is defined by, or equal to a number of entries in each index of an input tensor register used for a specific Single Instruction Multiple Data (SIMD) tensor operation, and all elements of A′ outside said G tensors are null. The system and method may further include executing kernel K on input I, by performing at least one computation of the SIMD tensor operation, having as operands elements of a tensor of the G tensors and corresponding elements of the B matrix.