Embedded AI Memory Architecture for Bandwidth and Power Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The limited memory bandwidth and high power consumption due to frequent data movement between compute units and memory in deep learning systems, particularly in energy-constrained environments, create significant bottlenecks in conventional Von-Neumann computer architecture, especially for artificial intelligence applications with large neural networks.
Innovation Solution
Integration of a memory device with a processor on the same chip, incorporating a DRAM interface and an embedded deep learning accelerator, enabling on-chip data movement and reducing latency and power consumption through a processing-in-memory (PIM) design.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is frequently moved between compute units and memory in conventional Von-Neumann architecture, then computations can be performed, but memory bandwidth is limited and power consumption increases
Solution Approach 1:
The patent merges memory and compute units into a unified processing-in-memory architecture where compute resources are embedded within the memory device. This eliminates the need for frequent data movement between separate memory and processor components, thereby reducing power consumption while maintaining computation throughput.
2Productivity
If data is frequently moved between compute units and memory, then computations can be performed, but memory bandwidth becomes a bottleneck
Solution Approach 1:
By integrating compute units within the memory device, the patent eliminates the external data movement bottleneck. The compute resources can directly access memory data through internal pathways, significantly improving effective memory bandwidth for AI workloads.
3Device complexity
If computations are performed using separate memory and processor components, then system architecture is simple, but data movement latency increases
Solution Approach 1:
The patent embeds compute units within the memory device structure, creating a processing-in-memory architecture. This integration dramatically reduces data movement latency by allowing computations to be performed directly on data within the memory array, eliminating the need for repeated data transfer between separate memory and processor components.
4Ease of manufacture
If conventional Von-Neumann architecture is used, then manufacturing is straightforward, but energy efficiency for AI applications deteriorates
Solution Approach 1:
The patent integrates compute resources with memory using standard semiconductor manufacturing processes, achieving processing-in-memory functionality without requiring fundamentally new manufacturing techniques. This approach improves energy efficiency for AI applications by eliminating data movement overhead while maintaining compatibility with existing manufacturing capabilities.
Data Source
AI summary
Systems, methods, and apparatus related to memory devices. In one approach, a memory device includes memories that store data received from a host device. The memory device includes a memory interface to the host device. The memory device further includes one or more processing devices to perform, using a portion of the data stored in one or more of the memories, computations for a neural network. An output of the neural network is stored in one of the memories. The memory device has a controller that controls memory access by the host device and the processing devices to avoid a conflict. The memory device communicates with the host device over the memory interface using a DRAM bus protocol. This communication includes sending the output of the neural network to the host device.