Neural Processor Rasterizer Data Splitting for Bandwidth Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network processing systems rely heavily on central processing units (CPUs), which lead to significant bandwidth consumption and increased power usage due to the extensive computing operations required for neural network operations.
Innovation Solution
A neural processor circuit with integrated rasterizer circuits at various components to manage and coordinate the processing of input data, splitting it into smaller units such as slices, tiles, and work units, and utilizing neural engines to perform multiply-accumulate operations efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If CPU is used to perform neural network operations, then ease of operation is improved, but power consumption increases
Solution Approach 1:
The patent divides the neural network processing system into distinct segments: a CPU that handles high-level operations and a specialized neural processing unit (NPU) that handles compute-intensive neural network operations. This segmentation allows the CPU to offload power-consuming tasks to the NPU, reducing overall power consumption while maintaining ease of operation through unified programming interfaces.
Solution Approach 2:
The patent introduces a neural processing unit as an intermediary component between the CPU and memory systems. This NPU acts as a mediator that receives neural network operation requests from the CPU, performs the computations using dedicated hardware accelerators, and returns results to the CPU, thereby reducing the CPU's power consumption while maintaining system ease of operation.
2Ease of operation
If CPU is used to perform neural network operations, then ease of operation is improved, but bandwidth consumption increases
Solution Approach 1:
The system segments data storage and processing functions by introducing a dedicated data buffer in the neural processing unit that can store input data, kernel data, and intermediate results. This segmentation reduces the need for continuous data transfers between main memory and the CPU, thereby reducing bandwidth consumption while maintaining ease of operation through unified data access interfaces.
Solution Approach 2:
The patent introduces a data buffer as an intermediary storage component within the neural processing unit. This buffer mediates data transfers by caching input data and kernel data locally, reducing the volume of data that must be transmitted over the system bus between the CPU and main memory, thus reducing bandwidth consumption while preserving ease of operation.
3Productivity
If input data is processed in large units, then processing efficiency is improved, but device complexity increases
Solution Approach 1:
The patent implements a hierarchical segmentation strategy where input data is divided into manageable chunks that are processed in parallel by multiple neural engines within the NPU. Each neural engine processes a segment of the data independently, achieving high processing efficiency through parallelism while keeping individual processing units relatively simple in structure.
Solution Approach 2:
The patent transitions from sequential processing to parallel processing by organizing multiple neural engines in a dimensional array structure. This dimensional organization allows the system to process large volumes of data simultaneously across multiple processing elements, achieving high processing efficiency without requiring each individual processing unit to be overly complex.
Data Source
AI summary
Embodiments of the present disclosure relate to splitting input data into smaller units for loading into a data buffer and neural engines in a neural processor circuit for performing neural network operations. The input data of a large size is split into slices and each slice is again split into tiles. The tile is uploaded from an external source to a data buffer inside the neural processor circuit but outside the neural engines. Each tile is again split into work units sized for storing in an input buffer circuit inside each neural engine. The input data stored in the data buffer and the input buffer circuit is reused by the neural engines to reduce re-fetching of input data. Operations of splitting the input data are performed at various components of the neural processor circuit under the management of rasterizers provided in these components.