Near-Memory Computing Architecture Sharing Inactive IMC as TCM
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Performing inference of a full ANN on resource-constrained platforms with real-time throughput requirements is computing-intensive and memory access-intensive, leading to high power consumption and inefficient hardware utilization due to inactive IMC devices.
Innovation Solution
An adaptive architecture that shares inactive IMC devices with other units as TCMs, utilizing a memory interface independent from the stream switch, and configures them via dedicated registers to optimize hardware utilization and reduce power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If In-Memory Computing (IMC) devices are used to perform data processing directly inside memory, then power consumption is reduced by avoiding excessive data traffic, but hardware utilization decreases because part of the IMC devices remain inactive at run-time
Solution Approach 1:
The patent enables IMC devices to perform multiple functions: they can operate as active computing units during neural network inference and as Tightly-Coupled Memory (TCM) for data reshape units when inactive. This multi-functionality is achieved through a memory interface that allows data reshape units to access inactive IMC devices, transforming them from idle resources into useful memory storage, thereby eliminating the 'dark silicon' problem while maintaining low power consumption characteristics
Solution Approach 2:
The system dynamically changes the operational state of IMC devices between compute mode and memory mode. By controlling the memory interface independently from the stream switch, the system can reconfigure inactive IMC devices to serve as TCM, effectively changing their functional parameter from computation to storage without physical modification, thus optimizing both power efficiency and hardware utilization
2Productivity
If dedicated hardware accelerators based on highly-parallel architectures are used to achieve real-time throughput, then computing performance is improved, but power consumption increases due to frequent memory accesses
Solution Approach 1:
The patent merges the computing function and memory function into a single IMC device. By performing data processing directly inside the memory array, the system eliminates the need for frequent data transfers between separate memory and processing units, thereby maintaining high throughput while significantly reducing power consumption associated with data movement
Solution Approach 2:
The memory interface acts as an intermediary that enables data reshape units to access inactive IMC devices as TCM. This intermediary mechanism allows the system to optimize data flow paths, reducing unnecessary memory accesses through the stream switch and lowering overall power consumption while maintaining throughput requirements
3Adaptability or versatility
If the geometry of convolutional kernels changes substantially from layer to layer, then adaptability to different neural network configurations is improved, but hardware utilization decreases because it is hard to perfectly map the CNN model on available resources
Solution Approach 1:
The system dynamically reconfigures the operational state of IMC devices based on the specific neural network layer being processed. When a layer requires fewer compute resources, previously active IMC devices are transitioned to memory mode to support data reshape operations, ensuring optimal hardware utilization across varying computational requirements of different network configurations
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
A hardware accelerator includes a plurality of functional circuits, a stream switch, one or more data reshape units coupled to the plurality of functional circuits via the stream switch to stream data to and from functional circuits of the plurality of functional circuits, and one or more In-Memory Computing (IMC) clusters coupled to the stream switch. In operation, inactive IMC devices of at least a subset of the one or more IMC clusters are accessible to at least a subset of the one or more data reshape units, via memory interface independent from the stream switch, to serve as at least part of Tightly-Coupled Memory (TCM) dedicated to at least one of the one or more data reshape units.