Distributed Neural Cores With Hierarchical Parallelism for Faster Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network computations are inefficient due to the lack of effective parallelism in processing, leading to suboptimal performance and resource utilization.
Innovation Solution
Implementing a network of distributed neural cores with parallel vector compute units that compute partial sums and output activations in parallel, allowing for efficient distribution of computation tasks across multiple cores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If neural network computations are performed using traditional sequential processing, then implementation is simpler, but computational efficiency and resource utilization are poor
Solution Approach 1:
The system segments the neural network computation into multiple independent neural cores, each capable of processing specific portions of the computation. Each neural core is further divided into vector compute units that handle specific vector operations. This segmentation enables parallel processing while maintaining manageable complexity at each level through hierarchical organization.
Solution Approach 2:
The patent introduces parallelism across multiple dimensions: temporal parallelism through pipelined stages within neural cores, spatial parallelism through multiple neural cores processing different data partitions, and vector parallelism through SIMD operations. This multi-dimensional parallelism transforms sequential processing into a highly parallel system without linearly increasing complexity.
2Speed
If multiple neural cores are used for parallel computation, then processing speed increases, but coordination and communication overhead increases
Solution Approach 1:
Multiple neural cores are merged into a unified parallel processing system where they collectively execute layers of the neural network. The system merges computation across cores by partitioning input data and accumulating results, reducing the need for complex inter-core communication while maintaining high processing speed through coordinated parallel execution.
Solution Approach 2:
The system performs preliminary partitioning of input data and assignment of computation tasks to specific neural cores before execution begins. This preliminary organization of work reduces runtime coordination overhead by pre-establishing data flow paths and computation assignments, allowing cores to execute independently with minimal synchronization requirements.
3Productivity
If vector compute units operate in parallel within each neural core, then computational throughput increases, but resource allocation complexity increases
Solution Approach 1:
Vector compute units within each neural core are designed as universal processing elements that can execute multiple types of vector operations (multiplication, accumulation, activation functions) through a unified architecture. This multi-functionality increases computational throughput while reducing resource allocation complexity by eliminating the need for specialized hardware for each operation type.
Solution Approach 2:
The vector compute units utilize parameter changes in data representation (e.g., different precision formats, vector lengths) to optimize computational throughput for different neural network layers and operations. By dynamically adjusting parameters rather than hardware configuration, the system achieves high throughput while keeping resource allocation management software-controlled and less complex.
Data Source
AI summary
Networks of distributed neural cores are provided with hierarchical parallelism. In various embodiments, a plurality of neural cores is provided. Each of the plurality of neural cores comprises a plurality of vector compute units configured to operate in parallel. Each of the plurality of neural cores is configured to compute in parallel output activations by applying its plurality of vector compute units to input activations. Each of the plurality of neural cores is assigned a subset of output activations of a layer of a neural network for computation. Upon receipt of a subset of input activations of the layer of the neural network, each of the plurality of neural cores computes a partial sum for each of its assigned output activations, and computes its assigned output activations from at least the computed partial sums.