Learnable Semi-Structured Sparsity for Low-Latency LLM Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large Language Models (LLMs) require substantial memory and computational resources, leading to high latency in real-world applications, and existing pruning techniques either necessitate significant retraining or fail to achieve efficient computational acceleration due to irregular sparse patterns.
Innovation Solution
Implement semi-structured pruning by learning a composite mask for each parameter block using a weighted average of candidate masks, optimizing for N:M sparsity through differentiable masks and machine learning techniques to select masks that maintain model performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If unstructured pruning is applied to remove individual parameters, then model compression is achieved, but computational acceleration is impeded due to irregular sparse patterns
Solution Approach 1:
The patent applies local quality by transitioning from global unstructured sparsity to local structured sparsity at the block level. Each parameter block is pruned independently with its own sparse pattern, allowing regularity within local regions while maintaining overall model compression. This local structured approach enables hardware-friendly computation patterns without requiring complete regularity across the entire model.
Solution Approach 2:
The patent segments the model parameters into discrete blocks and applies pruning at the block level rather than individually to all parameters. This segmentation creates manageable units with regular sparse patterns that can be processed efficiently by hardware, while the collection of segmented blocks achieves overall model compression.
2Productivity
If structured pruning is applied to physically eliminate substructures, then computational acceleration is achieved, but huge retraining efforts are required to recover network quality
Solution Approach 1:
The patent employs dynamic mask selection where different sparse masks are conditionally applied to different parameter blocks based on their importance and characteristics. This dynamic approach allows the model to adaptively choose between different sparsity patterns during inference, achieving acceleration while maintaining accuracy without requiring extensive retraining to recover from fixed structural pruning.
Solution Approach 2:
The patent changes the sparsity parameter at the block level rather than applying uniform structural pruning. By varying the sparsity pattern and density across different blocks, the model achieves computational acceleration in critical paths while preserving important parameters, reducing the need for retraining compared to coarse structured pruning.
3Reliability
If LLM scale is increased to improve generality and robustness, then model performance is improved, but memory footprint and inference latency increase
Solution Approach 1:
The patent introduces sparsity as an additional dimension for model optimization. Instead of simply scaling model size, the patent adds a sparsity dimension that allows larger models to be deployed with reduced memory footprint. By applying different sparsity levels across model dimensions, the patent achieves efficient deployment of large-scale models without linearly increasing memory requirements.
4Reliability
If LLM scale is increased to improve generality and robustness, then model performance is improved, but inference latency increases
Solution Approach 1:
The patent segments the inference computation into blocks with regular sparse patterns that can be processed efficiently. By organizing the large model into segmented blocks with structured sparsity, the patent enables parallel processing and reduces the computational burden per layer, thereby reducing overall inference latency while maintaining the robustness of large-scale models.
Data Source
AI summary
Apparatuses, systems, and techniques to losslessly compress neural networks via semi-structured sparsity. In at least one embodiment, a weighted average of candidate masks for semi-structured sparsity is learned for each parameter block of a neural network, and a composite mask is determined by selecting candidate masks based on the learned weighted averages. In at least one embodiment, computational resources required for inference are reduced, thereby contributing to more sustainable and environmentally friendly AI applications.