In-Memory Tensor Compute Architecture to Cut Data Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial intelligence applications face inefficiencies in energy usage and speed due to the need to transfer large amounts of data from memory to compute devices like GPUs for tensor operations, leading to increased complexity and data processing challenges.
Innovation Solution
A compute device with a scalable architecture that performs tensor operations directly in memory using dual in-line memory modules with compute logic units and media access circuitry, allowing for efficient data access and processing within the memory, reducing the need for data transfer through buses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is transferred from memory to GPU for compute operations, then compute operations can be performed, but energy consumption increases and processing speed decreases
Solution Approach 1:
The patent merges the memory subsystem and compute logic into an integrated memory compute device. The compute logic units are embedded within the memory architecture, allowing data to be processed in-place without physical transfer between separate memory and compute components. This integration eliminates the energy-consuming data movement and accelerates processing by reducing latency.
Solution Approach 2:
The patent introduces media access circuitry as an intermediary component that bridges memory media and compute logic units. This circuitry enables direct access to memory data by the compute units, eliminating the need for traditional bus-based data transfer through the host processor. The intermediary architecture allows efficient in-memory computing while maintaining memory integrity and access control.
2Adaptability or versatility
If data transfer through bus is used for AI compute operations, then general purpose processing is maintained, but efficiency and speed are hampered
Solution Approach 1:
The patent segments the compute device into multiple independent compute logic units, each capable of performing tensor operations autonomously. These units are distributed across the memory architecture, allowing parallel processing of AI workloads. The segmented architecture maintains versatility by supporting different compute operations while dramatically improving efficiency through concurrent data processing at the memory level.
3Speed
If data is kept in memory for processing, then data access is fast, but data must be transferred through bus consuming time and energy
Solution Approach 1:
The patent adds a new dimension to the traditional von Neumann architecture by embedding compute capabilities directly within the memory hierarchy. Instead of moving data along a single dimension (memory ↔ bus ↔ processor), the system allows compute operations to occur in-place within the memory structure. This dimensional change eliminates the data transfer bottleneck while preserving fast data access.
Data Source
AI summary
Technologies for providing a scalable architecture to efficiently perform compute operations in memory include a memory having media access circuitry coupled to a memory media. The media access circuitry is to access data from the memory media to perform a requested operation, perform, with each of multiple compute logic units included in the media access circuitry, the requested operation concurrently on the accessed data, and write, to the memory media, resultant data produced from execution of the requested operation.