Group-Sparse Neural Network Execution on CPUs Using Vectorized Kernels
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hardware systems lack efficient support for simultaneously executing neural networks that are both quantized and pruned, particularly on commodity CPUs, due to challenges with vectorized execution of sparse computations.
Innovation Solution
A method for creating neural network code that utilizes group-sparse kernels, where non-zero elements are grouped consecutively, allowing for efficient execution on CPUs using vectorized instructions like VNNI by pruning and quantizing neural networks to align with CPU architecture.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If neural networks are pruned and quantized to reduce computational resources, then memory usage and energy consumption decrease, but execution efficiency deteriorates on conventional CPUs
Solution Approach 1:
The patent segments the neural network computation into groups of consecutive non-zero elements (group-sparsity). By organizing sparse matrix operations into contiguous groups that align with CPU vector instruction widths, the system enables efficient parallel processing while maintaining the memory savings from pruning and quantization. This segmentation resolves the contradiction by making sparse computations compatible with conventional CPU architectures.
Solution Approach 2:
The patent changes the sparsity pattern parameter from arbitrary sparse distributions to structured group-sparsity patterns. By enforcing that non-zero elements appear in consecutive groups rather than random positions, the neural network maintains reduced memory usage while enabling vectorized execution on CPUs, thus improving execution efficiency without sacrificing the benefits of pruning and quantization.
2Adaptability or versatility
If conventional CPU vectorized instructions are used for sparse neural network execution, then hardware compatibility improves, but computational performance deteriorates due to inefficient sparse operation handling
Solution Approach 1:
The patent applies local quality by making specific regions of the sparse matrix (consecutive non-zero groups) have properties suitable for vectorized processing. Rather than requiring the entire sparse matrix to have a uniform structure, the method identifies and processes local contiguous regions that align with CPU vector instruction capabilities, enabling efficient execution on conventional hardware while maintaining overall sparsity benefits.
3Reliability
If arbitrary sparse patterns are maintained in pruned neural networks, then model accuracy is preserved, but execution efficiency deteriorates due to non-contiguous memory access patterns
Solution Approach 1:
The patent performs preliminary action by reorganizing the sparse matrix into group-sparsity format during the pruning and quantization phase, before execution. This pre-processing step groups consecutive non-zero elements together, establishing a memory access pattern that is optimized for vectorized operations. By preparing the data structure in advance, the system eliminates execution-time overhead from random memory access while preserving the computational savings from sparsity.
Data Source
AI summary
Creating neural network (NN) code may include for each row in a kernel matrix, finding the first non-zero element; and creating a loop including multiply and add instructions. On each iteration of the loop, the multiply and add instructions may be executed, and the position of the kernel matrix operand operated on by each multiply and add may be correlated to the loop iteration number. Instructions may be issued or created to be executed in the loop. A method may execute a NN by executing a loop including a series of multiply and add instructions to multiply a kernel matrix A by an input, such that on each iteration of the loop the series of multiply and add instructions are executed; and the position of the matrix A operand operated on by each multiply and add instruction in the series is correlated to the iteration number of the loop.