Pipelined PIM Accelerator for Vision Inference Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning inference applications face challenges in efficiently mapping neural networks onto hardware fabrics due to the slowdown of Moore's Law, leading to suboptimal power-performance in processing-in-memory (PIM) systems, particularly in handling multiple images simultaneously.
Innovation Solution
A pipelined processing-in-memory (PIM) system that employs inter-layer and intra-layer pipelining, allowing for concurrent processing of multiple images by generating multiple outputs in one clock cycle, utilizing a configurable hardware accelerator with mixed-signal PIMs to map machine learning applications efficiently, enabling stall-free and low-latency operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If traditional digital or PIM-based inference accelerators are used, then device complexity is reduced, but power-performance is suboptimal and throughput is limited
Solution Approach 1:
The system segments the neural network into multiple layers and processes them through a pipelined architecture with separate input interface, processing engine, and output interface. Each layer is processed independently through the pipeline, allowing concurrent processing of multiple images at different stages, thereby improving throughput while maintaining power efficiency.
Solution Approach 2:
The patent introduces a temporal dimension to the processing architecture by implementing pipelining across multiple clock cycles. Different images are processed at different stages of the pipeline simultaneously, transforming a sequential processing approach into a parallel temporal structure that increases throughput without proportionally increasing power consumption.
2Productivity
If multiple images are processed simultaneously, then throughput increases, but processing latency and complexity increase
Solution Approach 1:
The system performs preliminary actions by pre-loading input images into the input interface and pre-allocating processing resources for each layer before actual processing begins. The pipelined architecture is prepared in advance to accept multiple images, allowing them to flow through the pipeline with minimal idle time, thus reducing overall processing latency while maintaining high throughput.
Solution Approach 2:
The pipelined processing engine maintains continuous useful action by ensuring that at least one image is always being processed at each layer. The pipeline is designed to prevent idle cycles, with new images entering the pipeline as soon as previous images advance to the next layer, thereby maximizing throughput without significant increases in processing latency.
3Productivity
If pipelined processing is implemented, then throughput and power-performance improve, but device complexity and mapping difficulty increase
Solution Approach 1:
The processing engine is designed as a universal pipelined architecture that can process different types of neural network layers (convolutional, fully connected, etc.) through the same infrastructure. The input interface, processing engine, and output interface are multi-functional components that can handle various data formats and operations, reducing the need for specialized hardware for each layer type and simplifying the overall mapping process.
Solution Approach 2:
The system uses parameter changes to configure the pipelined architecture for different neural network models. By adjusting parameters such as pipeline depth, number of parallel processing units, and memory allocation, the system can optimize the mapping of different networks without requiring fundamental architectural changes, thereby managing device complexity while maintaining high throughput.
4Productivity
If batch processing of multiple images is performed, then throughput increases, but memory requirements and processing complexity increase
Solution Approach 1:
The batch processing approach segments the batch of images into individual processing streams that flow through the pipeline concurrently. Each image is processed independently through the same pipeline stages, allowing the system to handle multiple images without loading them all into memory simultaneously. This segmentation reduces peak memory requirements while maintaining high throughput through parallel processing.
Data Source
AI summary
A method of pipelining inference of a neural network, which includes an i-th layer (i being an integer greater than zero) and an (i+1)-th layer, includes processing, for a first input image, first i-th values of the i-th layer to generate first (i+1)-th values for the (i+1)-th layer, processing, for the first input image, the first (i+1)-th values of the (i+1)-th layer to generate output values, and concurrently with processing, for the first image, the (i+1)-th values, processing, for a second input image, second i-th values of the i-th layer to generate second (i+1)-th values.