Sparse Matrix-Vector Storage for Multicore Neural Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network processing systems face inefficiencies due to hardware limitations, such as pin-limited connectivity, computational complexity, and inefficient data storage and computation in multicore processors, which hinder their adoption in embedded devices.
Innovation Solution
The proposed solution involves hardware-aware mapping and partitioning of neural networks based on the target platform's limitations, leveraging sparsity and distributed scheduling to minimize unnecessary data transfers and computations, and using compressed data structures to fit within memory and processing capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If neural networks are implemented in general-purpose programming environments using commodity hardware, then hardware availability and software compatibility are improved, but processing efficiency and power consumption are worsened
Solution Approach 1:
The patent segments the neural network processing into distinct functional units including vector processing units, matrix processing units, and specialized compression hardware. Each unit handles specific operations independently, allowing parallel processing while maintaining compatibility with standard neural network architectures. This segmentation enables efficient utilization of commodity hardware resources.
Solution Approach 2:
The patent implements dynamic parameter adjustment mechanisms that modify processing parameters based on input data characteristics and hardware capabilities. Compression ratios, precision levels, and processing thresholds are adjusted in real-time to optimize the balance between processing efficiency and resource utilization on commodity hardware platforms.
2Quantity of substance
If compressed data structures are used to reduce memory requirements, then memory usage is improved, but data access complexity is worsened
Solution Approach 1:
The patent applies compression algorithms to neural network data structures during the network design and deployment phase, before the network is executed. Weight matrices and activation vectors are pre-compressed using specialized formats that preserve the necessary computational properties while reducing storage requirements. This preliminary compression eliminates the need for complex runtime decompression operations.
Solution Approach 2:
The patent introduces intermediary data structures that serve as bridges between compressed storage formats and processing requirements. These intermediary structures maintain the compressed representation while providing efficient access patterns for computational operations, eliminating the need for full decompression and reducing access complexity.
3Productivity
If sparsity is leveraged to reduce computational complexity, then processing speed is improved, but data storage requirements are worsened
Solution Approach 1:
The patent merges the storage of sparse data indicators with the actual data values in a unified compressed format. Instead of storing separate metadata about non-zero elements and their positions, the system combines this information with the data itself using run-length encoding and pattern-based compression techniques, reducing overall storage requirements while maintaining fast access to sparse structures.
Data Source
AI summary
Methods and apparatus for matrix and vector storage and operations are disclosed. Exemplary embodiments include a multicore processor, where each core includes processing hardware, local weights, global weights, working memory, and accumulator. Exemplary embodiments compress sparse neural network data structures based on actual, non-null, connectivity (rather than all possible connections). In some variants, the compression and reduction in complexity is sized to fit within the memory footprint and processing capabilities of a core. The exemplary compression schemes represent sparse matrices with links to compressed column data structures, where each compressed column data structure only stores non-null entries to optimize column-based lookups of non-null entries. Similarly, sparse vector addressing skips nulled entries when performing vector-specific non-null multiply-accumulate operations.