Near-Memory Computing Reusing Inactive IMC Clusters as TCM
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Performing inference of a full ANN on resource-constrained platforms with real-time throughput requirements is computing-intensive and power-consuming due to frequent memory accesses, and IMC devices are sub-par in performance and inefficiently utilized due to geometry changes in convolutional kernels.
Innovation Solution
An adaptive architecture that shares inactive IMC devices with other units as TCMs, utilizing dedicated configuration registers to select IMCs for TCM, and implements a neural network mapping tool to avoid collisions and data corruption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If IMC devices are used to perform data processing directly inside memory, then power consumption is reduced by avoiding excessive data traffic, but hardware utilization decreases due to geometry changes in convolutional kernels making part of IMC devices inactive
Solution Approach 1:
The patent enables IMC devices to perform multiple functions: they can operate as active computing units for convolution operations or be dynamically allocated as TCM for data buffering. This multi-functionality allows the same hardware resource to adapt its role based on runtime requirements, improving both energy efficiency and utilization.
Solution Approach 2:
The system implements dynamic allocation where the controller can reconfigure IMC devices between compute mode and memory mode based on the specific layer requirements. This dynamic adaptability ensures that IMC devices are neither left idle nor overworked, optimizing both power consumption and hardware utilization across different network layers.
2Productivity
If multiple IMC devices are allocated to accommodate convolutional kernel geometries, then computing performance is improved, but hardware complexity increases due to difficulty in perfectly mapping CNN model on available resources
Solution Approach 1:
The patent segments the IMC cluster into multiple independently controllable devices, each capable of being configured for specific kernel geometries. This segmentation allows flexible mapping of different CNN layers to appropriate IMC devices, simplifying the overall mapping complexity while maintaining high computing performance.
Solution Approach 2:
The controller acts as an intermediary that manages the mapping between CNN layers and IMC devices. It handles the complexity of matching kernel geometries to device configurations, shielding the rest of the system from this complexity while enabling optimal computing performance through intelligent resource allocation.
3Speed
If dedicated TCM is provided for data reshape units, then data streaming efficiency is improved, but hardware resource waste occurs when IMC devices remain inactive
Solution Approach 1:
The patent implements a resource recovery mechanism where IMC devices that finish their compute tasks are immediately reallocated as TCM for data reshape operations. This continuous recycling of hardware resources eliminates idle time and prevents resource waste, while ensuring dedicated TCM availability for data streaming when needed.
Solution Approach 2:
The system maintains continuous useful action by seamlessly transitioning IMC devices from compute mode to memory mode without idle periods. The dynamic allocation ensures that every IMC device is continuously engaged in either computation or data buffering, eliminating resource waste while maintaining efficient data streaming throughput.
Data Source
AI summary
A hardware accelerator includes a plurality of functional circuits, a stream switch, one or more data reshape units coupled to the plurality of functional circuits via the stream switch to stream data to and from functional circuits of the plurality of functional circuits, and one or more In-Memory Computing (IMC) clusters coupled to the stream switch. In operation, inactive IMC devices of at least a subset of the one or more IMC clusters are accessible to at least a subset of the one or more data reshape units, via memory interface independent from the stream switch, to serve as at least part of Tightly-Coupled Memory (TCM) dedicated to at least one of the one or more data reshape units.