Streaming Convolution Buffering for Low-Memory Neural Processors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network processing systems rely heavily on CPUs, leading to significant bandwidth consumption and increased power usage due to extensive computing operations.
Innovation Solution
A neural processor circuit with a neural engine and data processor circuit performs operations in a streaming manner, utilizing tensor buffers to store and manage input and output tensors efficiently, allowing parallel execution of multiple layers of convolutions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If CPU and main memory are used to instantiate and execute machine learning systems, then ease of configuration is improved, but bandwidth consumption and power consumption increase significantly
Solution Approach 1:
The system is divided into multiple tensor buffers (first tensor buffer, second tensor buffer, third tensor buffer, fourth tensor buffer) that are distributed across different memory banks. This segmentation allows parallel access to different portions of input tensors by multiple neural processing units simultaneously, reducing the burden on any single CPU core and memory bank, thereby lowering overall power consumption while maintaining configurability.
Solution Approach 2:
The patent introduces a new dimension of parallelism by organizing tensor buffers across multiple memory banks that can be accessed simultaneously by different neural processing units. This dimensional expansion from sequential CPU processing to parallel distributed memory access reduces bandwidth consumption per component and overall power usage while preserving ease of configuration through software-controlled buffer allocation.
2Reliability
If multiple layers of convolutions are performed on input data, then processing completeness is improved, but memory footprint and processing time increase
Solution Approach 1:
The system performs preliminary actions by pre-fetching and storing portions of input tensors in tensor buffers before they are needed by neural processing units. The data control circuit prepares data in advance by dividing input tensors into multiple portions and distributing them to appropriate buffers, enabling neural processing units to immediately begin computation without waiting for data retrieval, thus reducing overall processing time while maintaining complete multi-layer processing.
Solution Approach 2:
Tensor buffers serve as intermediaries between main memory and neural processing units. The data control circuit acts as a mediator that manages data flow between memory banks and processing units, coordinating the transfer of tensor portions to appropriate buffers. This intermediary layer enables efficient memory management and reduces the memory footprint required for complete multi-layer convolution processing by allowing selective loading of only necessary data portions.
3Productivity
If partial input tensors are stored in tensor buffers for streaming processing, then processing speed is improved, but data management complexity increases
Solution Approach 1:
The data control circuit performs multiple functions: it divides input tensors into portions, distributes them to appropriate tensor buffers, monitors buffer status, and coordinates access by neural processing units. This universal data management component handles all aspects of tensor buffer management through a unified control mechanism, reducing the apparent complexity for individual processing units while enabling high-speed streaming processing. The same control circuit manages both first and second tensor buffers using similar protocols.
Solution Approach 2:
The data control circuit implements feedback mechanisms by monitoring the status of tensor buffers and adjusting data transfer operations accordingly. When neural processing units complete processing of data from one buffer, the control circuit receives feedback and initiates transfer of the next data portion from main memory to the appropriate buffer. This feedback-driven approach automates data management tasks, reducing manual intervention complexity while maintaining high processing speed through continuous data flow.
Data Source
AI summary
Embodiments relate to streaming operations in a neural processor circuit that includes a neural engine circuit and a data processor circuit. The neural engine circuit performs first operations on a first input tensor of a first layer to generate a first output tensor, and second operations on a second input tensor of a second layer at a higher hierarchy than the first layer, the second input tensor corresponding to the first output tensor. The data processor circuit stores a portion of the first input tensor for access by the neural engine circuit to perform a subset of the first operations and generate a portion of the first output tensor. The data processor circuit stores the portion of the first output tensor for access by the neural engine circuit as a portion of the second input tensor to perform a subset of the second operations.