Multiple Accelerators for Neural Network Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network systems face inefficiencies in processing large parameter data sets, often requiring external memory and complex data management due to the limitations of individual accelerator units.
Innovation Solution
Implementing a system with multiple accelerators managed by a controller unit, where each accelerator has internal storage, allowing for sub-network creation by grouping or dividing layers, enabling efficient data processing without external memory and optimizing computation through data aggregation and distribution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single accelerator unit is used for neural network processing, then device complexity is reduced, but processing efficiency and productivity deteriorate due to inability to handle large parameter data sets
Solution Approach 1:
The patent divides the neural network processing system into multiple accelerator units, where each accelerator is responsible for specific layers or sub-networks. This segmentation allows parallel processing of different data sets, improving overall productivity while keeping individual accelerator complexity manageable.
Solution Approach 2:
The patent introduces a new dimension of organization by grouping accelerator units into hierarchical structures with controller units that manage multiple accelerators. This dimensional organization enables efficient handling of large parameter data sets through distributed processing without requiring proportional increases in individual accelerator complexity.
2Quantity of substance
If external memory is used to store parameter data, then storage capacity is increased, but data management complexity and processing time increase
Solution Approach 1:
The patent implements nested storage structures where parameter data is organized in hierarchical memory levels. Frequently accessed parameters are stored in faster, smaller memory close to the accelerator, while less frequently accessed data resides in larger, slower external memory. This nesting provides both adequate storage capacity and fast access for critical operations.
Solution Approach 2:
The system performs preliminary actions by pre-loading parameter data into accelerator-attached memory before processing begins. Data is fetched and organized in advance according to the processing schedule, eliminating wait times during actual computation and reducing overall processing time.
3Productivity
If parameter data is distributed across multiple accelerators, then processing capacity is improved, but data management and coordination complexity increase
Solution Approach 1:
The patent introduces controller units as intermediaries that manage data flow and coordination between multiple accelerator units. These controllers handle the complexity of distributed data management, allowing individual accelerators to focus on computation while the intermediary manages the complexities of data distribution and synchronization.
Solution Approach 2:
The controller units serve multiple functions: they manage data distribution to accelerators, collect results, coordinate processing schedules, and handle error management. This multi-functionality consolidates complex data management tasks into dedicated units, reducing the overall system complexity burden.
Data Source
AI summary
Technologies are described for multiple accelerators for a neural network, and methods thereof. In an example implementation, a neural network can be mapped to a system comprising a control unit and multiple accelerators, where the controller unit controls each accelerator's behavior, sends data to and receives data from each accelerator through the interconnections. Sub-networks may be created by grouping several network layers or dividing a network layer into multiple sub-layers depending on data to be processed and memory capacity of each accelerator. Accelerators have internal storage, thus, do not require external memory.