Global Scheduler for Neural Inference Processor Weight Distribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural inference processors face inefficiencies in runtime scheduling and data management, particularly in distributing synaptic weights and activation data across multiple cores, which hampers the performance of neural network computations.
Innovation Solution
A neural inference processor architecture is introduced, featuring a global scheduler and local controllers within each core, which loads synaptic weights from a neural network model memory, applies them to input activations, and stores output activations, optimizing the pipelined processing of neural network operations across multiple cores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a global scheduler is introduced to distribute synaptic weights across multiple cores, then the throughput of neural inference processing is improved, but the device complexity increases
Solution Approach 1:
The system divides the neural network model into multiple weight blocks that can be distributed across multiple cores. Each core processes a portion of the neural network computation independently, allowing parallel processing while maintaining manageable complexity through modular organization of weights and computations.
Solution Approach 2:
A global scheduler acts as an intermediary component that coordinates weight distribution and activation data flow between memory and multiple cores. This mediator manages the complexity of distributed data management by centralizing scheduling functions while enabling parallel core operations.
2Speed
If synaptic weights are distributed across multiple cores, then the processing speed is improved, but the data management complexity increases
Solution Approach 1:
The system pre-loads synaptic weight blocks into local memory buffers of each core before computation begins. This preliminary action ensures that weights are readily available during processing, eliminating runtime data management overhead and simplifying the data flow while maintaining high processing speed.
Solution Approach 2:
The architecture separates weight storage and activation data into distinct memory spaces, with weights stored in dedicated neural network model memory and activations in separate activation memory. This dimensional separation simplifies data management by organizing data access patterns along different memory dimensions, reducing management complexity while enabling parallel processing.
Data Source
AI summary
Neural inference processors are provided. In various embodiments, a processor includes a plurality of cores. Each core includes a neural computation unit, an activation memory, and a local controller. The neural computation unit is adapted to apply a plurality of synaptic weights to a plurality of input activations to produce a plurality of output activations. The activation memory is adapted to store the input activations and the output activations. The local controller is adapted to load the input activations from the activation memory to the neural computation unit and to store the plurality of output activations from the neural computation unit to the activation memory. The processor includes a neural network model memory adapted to store network parameters, including the plurality of synaptic weights. The processor includes a global scheduler operatively coupled to the plurality of cores, adapted to provide the synaptic weights from the neural network model memory to each core.