In-Memory Compute Offloading for Irregular Memory Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional CPU and GPU architectures struggle with irregular memory access patterns in applications like graph analytics and AI, leading to unpredictable delays and high energy consumption due to inefficient data movement between CPU and off-chip memory.
Innovation Solution
A computing system that automatically offloads sequences of machine instructions and memory accesses to a secondary computing device, such as an in-memory compute (IMC) or compute near memory (CNM) device, using trainable machine learning components to recognize patterns and route operations intelligently, thereby reducing latency and energy consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If data is moved between CPU and off-chip memory for irregular memory access patterns, then memory operations can be performed, but energy consumption increases significantly and unpredictable delays occur
Solution Approach 1:
The patent introduces an intermediary system consisting of a neural network processor and memory management unit that acts as a mediator between the CPU and off-chip memory. This intermediary learns and predicts irregular memory access patterns, pre-fetching data before the CPU needs it, thereby reducing the frequency and volume of data movements between CPU and off-chip memory, which directly addresses the high energy consumption issue while maintaining memory operation efficiency
Solution Approach 2:
The system performs preliminary actions by using the neural network to predict future memory access patterns and pre-fetching required data into faster memory buffers before the CPU actually needs it. This preliminary data preparation reduces the need for urgent data movements later, thereby lowering energy consumption and eliminating unpredictable delays in memory operations
2Adaptability or versatility
If manual mapping is used to configure IMC/CNM devices for computational kernels, then computing operations can be performed near memory, but flexibility is reduced and adaptability to diverse applications becomes difficult
Solution Approach 1:
The patent implements dynamic configuration by replacing static manual mapping with a neural network-based adaptive system. The neural network continuously learns from actual runtime memory access patterns and dynamically adjusts the mapping between computational kernels and IMC/CNM devices. This dynamic adaptation enables the system to automatically optimize its configuration for diverse applications without requiring complex manual setup, thereby improving flexibility while managing complexity through automation
Solution Approach 2:
The system provides self-service by enabling IMC/CNM devices to automatically configure themselves through the neural network's learned patterns. Instead of requiring external manual mapping, the system autonomously optimizes its own configuration by analyzing memory access patterns and automatically adjusting device mappings, which greatly enhances adaptability to diverse applications while reducing configuration complexity
3Adaptability or versatility
If static compilation is used for application execution, then code can be compiled in advance, but adaptability to dynamic workload characteristics and memory access patterns is lost
Solution Approach 1:
The patent transforms the static compilation model into a dynamic runtime adaptation model. The neural network continuously learns from actual workload characteristics and memory access patterns during execution, dynamically adjusting its predictions and optimizations. This dynamic approach enables the system to adapt to changing workload conditions in real-time, gaining adaptability while the incremental learning process avoids the overhead of complete re-compilation, thus managing time loss effectively
Data Source
AI summary
An example system includes memory; a central processing unit (CPU) to execute first operations; in-memory execution circuitry in the memory; and detector software to cause offloading of second operations to the in-memory execution circuitry, the in-memory execution circuitry to execute the second operations in parallel with the CPU executing the first operations.