Memory Processing Units for Low-Latency Feature Vector Summation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The von Neumann bottleneck in conventional computer architecture limits processing speeds due to throughput limitations in data transfer from memory to the processor, particularly in memory-intensive processes such as neural networks and machine learning algorithms, leading to inefficient use of clock cycles and computational bandwidth.
Innovation Solution
The implementation of memory processing modules (MPMs) with processing elements and dedicated memory banks on a common substrate, allowing for local data processing and reduced latency through proximity of processors to memory, along with advanced error correction and data handling mechanisms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data transfer from memory to processor is performed using conventional computer architecture, then the system can operate with standard von Neumann architecture, but the throughput is limited causing bottleneck in processing speeds
Solution Approach 1:
The patent merges the processor and memory into a single integrated unit where processing elements are directly coupled to memory banks on the same substrate. This eliminates the separate von Neumann architecture bottleneck by combining data storage and processing functions into one unified structure, allowing simultaneous access and computation without external data transfer delays.
Solution Approach 2:
The integrated memory device is divided into multiple independent memory banks, each associated with specific processing elements. This segmentation allows parallel data access and processing operations to occur simultaneously across different banks, increasing overall throughput and eliminating the single-bottleneck limitation of conventional architectures.
2Productivity
If memory-intensive processes are performed using conventional architecture, then the system can handle tasks like neural networks and machine learning, but clock cycles are consumed by reading and writing from memory rather than computation
Solution Approach 1:
By integrating processing elements directly within the memory device, the patent eliminates the time-consuming data transfer phase between separate memory and processor units. Processing elements can immediately access and compute on data stored in their associated memory banks, converting clock cycles that were previously lost to memory access into productive computation time.
Solution Approach 2:
The processing elements serve themselves by directly accessing data in their locally coupled memory banks without requiring external memory controllers or data transfer protocols. This self-service capability eliminates the overhead of conventional memory access sequences and allows continuous computation without idle clock cycles.
3Power
If the computational bandwidth of the processor is larger than the bandwidth of the buses used to access memory, then the processor can perform computations faster, but the bus bandwidth becomes the limiting factor
Solution Approach 1:
The patent merges the high-bandwidth computational resources with equally high-bandwidth memory access by placing processing elements directly on the memory substrate. This eliminates the bus bandwidth limitation because data no longer needs to travel through external buses—the processing elements access memory through direct internal connections with comparable bandwidth to the computational resources.
Data Source
AI summary
Disclosed embodiments include a computational memory system. The computational memory system includes at least one computational memory chip including one or more processor subunits and one or more memory banks formed on a common substrate. The at least one computational memory chip is configured to store one or more portions of an embedding table in the one or more memory banks, the embedding table including one or more feature vectors. The one or more processor subunits are configured to receive a sparse vector indicator from a host external to the at least one computational memory chip and, based on the received sparse vector indicator and the one or more portions of the embedding table, generate one or more vector sums.