Ultra-Pipelined Machine Learning Accelerator for Low-Latency PIM Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning applications face inefficiencies in mapping onto processing-in-memory (PIM) hardware fabrics, leading to slow processing times and suboptimal power-performance, particularly in neural network inference tasks.
Innovation Solution
A pipelined approach is employed for mapping machine learning applications onto PIM accelerators, utilizing inter-layer and intra-layer pipelining to concurrently process multiple layers and images, with a configurable hardware accelerator based on mixed-signal PIMs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional mapping approaches are used for ML applications onto PIM hardware, then device complexity is reduced, but processing speed and power-performance deteriorate
Solution Approach 1:
The patent segments the neural network computation into multiple pipeline stages distributed across different PIM subarrays. Each subarray handles specific layers or operations, allowing concurrent processing of different data batches at various computation stages. This segmentation enables higher throughput while maintaining manageable complexity through modular architecture.
Solution Approach 2:
The patent introduces temporal dimension through pipelining, allowing multiple layers to be processed in parallel across different time stages. By organizing computation in a multi-stage pipeline rather than sequential processing, the system achieves higher productivity without proportionally increasing device complexity.
2Loss of time
If sequential processing is used for neural network layers, then device complexity is minimized, but processing time and latency increase
Solution Approach 1:
The patent implements preliminary action by pre-configuring multiple PIM subarrays with different layer mappings before computation begins. Data is pre-loaded into the pipeline stages, and computation proceeds through predetermined pipeline stages without dynamic reconfiguration, reducing latency while keeping the control logic simple and fixed.
Solution Approach 2:
The patent ensures continuity of useful action by maintaining an always-active computation pipeline where each stage continuously processes data. Rather than completing one layer sequentially before moving to the next, the pipeline keeps all stages occupied with computation operations, eliminating idle time and reducing overall latency.
3Power
If conventional PIM architectures are used, then ease of manufacture is maintained, but power-performance ratio becomes suboptimal
Solution Approach 1:
The patent implements universality by designing PIM subarrays that can be configured to handle different neural network layers and operations. The same physical hardware structure serves multiple functions by reconfiguring data flow patterns and computational mappings, achieving high power-performance without requiring specialized hardware for each layer type.
Solution Approach 2:
The patent introduces dynamic configurability through programmable mapping that allows the system to adapt to different neural network architectures and workloads. This dynamic reconfiguration capability enables optimization of power-performance for various applications while maintaining a standardized hardware platform that is relatively easy to manufacture.
Data Source
AI summary
A method of pipelining inference of a neural network, which includes an i-th layer (i being an integer greater than zero), an (i+1)-th layer, and an (i+2)-th layer, includes processing a first set of i-th values of the i-th layer to generate (i+1)-th values for the (i+1)-th layer, determining a quantity of the (i+1)-th values as being sufficient for processing, and in response to the determining, processing the (i+1)-th values to generate an output value for the (i+2)-th layer while concurrently processing a second set of i-th values of the i-th layer.