HW-Aware Sparsity Pattern for Deep Learning Inference Acceleration

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Deep neural networks (DNNs) face challenges during industrial deployment due to their high computational cost of inference, primarily attributed to computation-intensive operations like General Matrix Multiple (GEMM) and convolution.

Innovation Solution

A hardware-aware sparsity pattern is introduced to accelerate deep learning inference, which involves determining a sparsity pattern based on the register width specified by the hardware's Instruction Set Architecture (ISA). This pattern specifies a block size and sparsity ratio for block-wise sparsification of weight matrices, allowing for efficient compression and inference on hardware units with SIMD capabilities.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If block-wise sparsification with HW-aware sparsity pattern is applied, then inference speed is improved (5×-10× speedup), but computational accuracy may deteriorate

Engineering Contradiction:

Improveinference speedVSAvoidcomputational accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The weight matrix is divided into multiple blocks, and sparsification is applied at the block level rather than element level. Each block can be independently pruned while maintaining the overall matrix structure, enabling hardware-efficient computation with reduced accuracy loss

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The sparsity ratio and block size are optimized as可调 parameters to achieve the best trade-off between inference speed and accuracy. The HW-aware sparsity pattern adapts parameters based on hardware capabilities (register width, SIMD lane count) to maximize performance while maintaining computational precision

Inventive Principle:

Principle #35Parameter changes

2Loss of time

If higher sparsity ratio is used to reduce computational cost, then inference time is reduced, but model accuracy deteriorates

Engineering Contradiction:

Improveinference timeVSAvoidmodel accuracy

Core Design Contradiction:

Loss of timeVSMeasurement precision

Solution Approach 1:

The sparsity pattern is made dynamic and adaptive rather than fixed. The system can adjust the sparsity ratio and block configuration based on hardware capabilities and workload characteristics, allowing optimal balance between inference time and accuracy for different scenarios

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

Sparsification is performed during the training phase rather than at inference time. The model weights are pre-pruned to match the HW-aware sparsity pattern, so that the inference engine can directly utilize the sparse structure without additional computation or accuracy degradation

Inventive Principle:

Principle #10Preliminary action

3Productivity

If unstructured sparsity pattern is applied to achieve higher sparsity ratio, then computational cost is reduced, but hardware compatibility deteriorates

Engineering Contradiction:

Improvecomputational efficiencyVSAvoidhardware compatibility

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The sparsity pattern introduces asymmetry at the block level while maintaining symmetry at the element level within blocks. This asymmetric block structure enables higher sparsity ratios while remaining compatible with symmetric SIMD hardware operations

Inventive Principle:

Principle #4Asymmetry

Solution Approach 2:

The HW-aware sparsity pattern is designed to be universally compatible with various SIMD hardware architectures. By adapting to different register widths and lane counts, the same sparsification approach can be applied across different hardware platforms without requiring architecture-specific implementations

Inventive Principle:

Principle #6Universality (Multi-functionality)

4Productivity

If existing sparsity pattern (e.g., 2-in-4) is used with hardware support, then inference speed is improved, but sparsity ratio is limited

Engineering Contradiction:

Improveinference speedVSAvoidsparsity ratio

Core Design Contradiction:

ProductivityVSQuantity of substance

Solution Approach 1:

The patent moves from element-level sparsification to block-level sparsification, adding a new dimension to the sparsity structure. This block-wise approach enables much higher sparsity ratios (e.g., 75% or more) compared to element-level patterns like 2-in-4, while still maintaining hardware compatibility through structured blocking

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Data Source

PatentUS20250045586A1Method and apparatus for accelerating deep leaning inference based on HW-aware sparsity pattern

Publication Date: 2025.02.06 INTEL CORP

AI summary

The application provides a method and apparatus for accelerating deep learning inference based on a HW-aware sparsity pattern. The method may include determining a hardware-aware sparsity pattern based on a register width specified by an ISA of a hardware unit for implementing the DNN for deep learning inference, the sparsity pattern specifying a block size and a sparsity ratio for block-wise sparsification of a weight matrix of an operator in the DNN; performing the block-wise sparsification for the weight matrix based on the sparsity pattern to obtain a sparse weight matrix, during a training process of the DNN; compressing the sparse weight matrix into a concentrated weight matrix by removing all-zero blocks from the sparse weight matrix; and generating a mask to indicate an index of each row of non-zero blocks in the sparse weight matrix to enable extraction of corresponding elements from an activation matrix of the operator.