Neural Network Processor Fetch Unit for Memory Access Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network processors face challenges in achieving high processing rates at a low cost, particularly in deep learning applications where the increasing number of nodes leads to inefficiencies in memory access and data transmission, resulting in high costs and energy consumption.
Innovation Solution
A calculation processing device with a fetch unit that reads data from multiple memory slices and distributes it to operation units through a network of routers, utilizing a software topology and data processing mapping tables to optimize data flow and reduce waiting times, allowing for efficient data processing and multicasting of tensor data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the number of nodes in the neural network is increased to improve processing capability, then the calculation precision and processing capability are improved, but the memory access time and energy consumption increase
Solution Approach 1:
The memory system is segmented into multiple memory slices (first memory slice, second memory slice, etc.) that can be accessed in parallel. The fetch unit is also segmented into multiple fetch buffers corresponding to different memory slices, enabling simultaneous data retrieval from multiple memory locations, thus reducing overall memory access time while supporting increased node count
Solution Approach 2:
Data is pre-fetched from memory slices into fetch buffers before being needed by the operation units. The fetch unit proactively loads data into buffers based on predicted or scheduled requirements, so that when operation units need the data, it is already available in the buffer, eliminating waiting time and reducing memory access delays
2Measurement precision
If the number of nodes is increased to improve processing capability, then the calculation precision is improved, but the energy consumption increases
Solution Approach 1:
The system is divided into multiple independent memory slices and fetch buffers that can operate in parallel. This segmentation allows the system to process data from different nodes simultaneously without requiring all data to be loaded into a single large buffer, reducing the total energy required for data management while maintaining support for increased node count
Solution Approach 2:
The fetch unit continuously supplies data to operation units by maintaining multiple active fetch buffers that are constantly being filled from memory slices. This continuous data supply ensures that operation units never idle, maximizing computational throughput and energy efficiency, as the system maintains steady-state operation rather than experiencing start-stop cycles
3Productivity
If data is stored in multiple memory slices to reduce memory access time, then the processing rate is improved, but the device complexity increases
Solution Approach 1:
Each memory slice and its corresponding fetch buffer are designed with identical, standardized interfaces and control logic. This universal design allows the system to scale by simply adding more slices and buffers without increasing the complexity of individual components, as each unit performs the same function and can be controlled by the same fetch unit architecture
Solution Approach 2:
The system uses a centralized fetch unit that proactively manages data movement from memory slices to fetch buffers before computation is needed. This preliminary data preparation centralizes the complexity of multi-memory-management in a single control unit, allowing the memory slices themselves to remain simple, independent storage units while still achieving high processing rates through coordinated data supply
Data Source
AI summary
The present specification discloses an arithmetic processing device which has a high processing rate with a low cost. The arithmetic processing device according to the present specification is an arithmetic processing device comprising a fetch unit which reads data required for a calculation for performing processing of a neural network from a memory, and provides the data to an arithmetic unit. The fetch unit may include: a fetch buffer into which data stored in each of data memory slices is fetched; and an interface controller for assigning a node ID corresponding to each of the data memory slices to the data fetched into the fetch buffer, and controlling a timing of outputting the fetched data according to the node ID.