Neural Weight Memory Alignment for Lockstep Core Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks with near-zero magnitude parameters face challenges in hardware acceleration due to unstructured sparsity, leading to high pruning rates and irregularities in weight distribution that hinder efficient processing on multiple parallel cores.
Innovation Solution
A system and method for neural network processing that includes processor cores with weight memories, using weight mask bits to indicate pruned or non-pruned weights, and a weight depruner to align and modify weights for efficient processing, ensuring lockstep operation and reduced memory requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If unstructured sparsity is used to achieve high pruning rate, then compressibility is improved, but processing efficiency on parallel cores deteriorates due to irregularities in weight distribution
Solution Approach 1:
The weight distribution is segmented into multiple sections that are distributed across different memory locations. Each processor core is assigned a specific section to process, enabling parallel processing while maintaining consistent data patterns. This segmentation resolves the contradiction by organizing sparse weights into manageable units that can be efficiently distributed to multiple cores.
Solution Approach 2:
Different sections of weights are assigned to different processor cores based on their local characteristics and pruning patterns. Each core processes weights with similar sparsity characteristics in its assigned section, optimizing processing efficiency while maintaining overall compression benefits. This local quality approach ensures that each core works on homogeneous data patterns.
2Productivity
If weights are processed on multiple parallel cores, then processing throughput is improved, but processing stalls increase due to misalignment in weight sections
Solution Approach 1:
Weight sections are pre-aligned and organized before distribution to processor cores. The system performs preliminary sorting and positioning of weight data so that each core receives sequentially aligned sections. This preliminary action prevents processing stalls during execution by ensuring all cores operate on synchronously available data.
Solution Approach 2:
The weight memory is structured to provide equipotential access patterns to all processor cores. Each core receives weight sections of equal size and with equivalent processing complexity, eliminating variability in processing completion times. This equipotential design ensures all cores progress through weight processing at the same rate, preventing stalls.
3Quantity of substance
If pruned weights are stored in memory, then memory usage is reduced through compression, but memory access complexity increases due to irregular weight patterns
Solution Approach 1:
The compressed weight memory is segmented into structured sections that correspond to specific processor core assignments. Each segment maintains a consistent format and access pattern, simplifying memory access while preserving compression benefits. This segmentation transforms irregular memory patterns into organized, predictable access structures.
Solution Approach 2:
A weight management layer acts as an intermediary between the compressed memory storage and processor cores. This intermediary layer handles the complexity of decoding and aligning compressed weight sections, presenting simplified access interfaces to each core. The intermediary absorbs memory access complexity while maintaining compression efficiency.
Data Source
Figure 1A~1C
Figure 2
Figure 3
AI summary
A neural network processing system and a method of generating weights for a neural network processing system is described. The system includes a plurality of processor cores coupled to respective weight memories which store neural network weights. The neural network weights are stored as a plurality of weight mask bits, each weight mask bit indicating whether a corresponding weight is a pruned weight or a non-pruned weight and a plurality of non-pruned weights. At least one of the non-pruned weights has a pruned weight value. Non-pruned weights with a pruned weight value may be selectively added after initial pruning to equalize memory section size, word align memory sections or to ensure processing stalls (hiccups) occur in the same cycle. The resulting pruned weight sets may be used with neural processor accelerators operating in lock step.