Cross-Point Memory Tensor Processing via In-Memory Broadcasting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The energy-intensive data transfer between memory and processor in compute devices, particularly for tensor operations, acts as a bottleneck in the speed and efficiency of artificial intelligence applications like neural networks, due to the need for frequent matrix data movement across the bus.
Innovation Solution
A compute device architecture with a three-dimensional cross-point memory and media access circuitry that allows for parallel data access, utilizing scratch pads for low-latency operations and broadcasting data within memory to reduce bus usage, enabling tensor operations to be performed efficiently within the memory itself.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If matrix data is transferred between memory and processor through a bus for tensor operations, then the operations can be performed with standard architecture, but energy consumption increases and processing speed decreases due to frequent data movement
Solution Approach 1:
The patent combines memory storage and tensor processing functions into a single integrated memory device. The media access circuitry within the memory performs tensor operations directly on stored matrix data, eliminating the need for separate processor units and reducing data transfer requirements between components.
Solution Approach 2:
The patent transitions from traditional von Neumann architecture (separate memory and processor) to an in-memory computing paradigm. By embedding processing capabilities within the memory device itself, it creates a new operational dimension where computation occurs at the location of data storage, fundamentally changing the data flow architecture.
2Productivity
If matrix data is transferred through the bus between memory and processor, then standard compute architecture can be used, but the frequency of data movement creates a bottleneck reducing overall efficiency
Solution Approach 1:
The patent extracts the tensor processing function from the external processor and relocates it directly into the memory device. The media access circuitry within memory now performs tensor operations on matrix data stored in the memory media, eliminating the need for repeated data transfers to and from an external processor.
Solution Approach 2:
The media access circuitry acts as an intermediary between the memory media and external processors. It retrieves matrix data from memory media, performs tensor operations internally, and returns results, thereby eliminating the need for continuous data shuttling between external memory and processor through the bus.
3Speed
If data is accessed sequentially through the bus for tensor operations, then simple architecture can be maintained, but processing speed is limited by the sequential nature of data transfer
Solution Approach 1:
The patent segments the memory device into multiple partitions, each capable of independent tensor operations. This parallel architecture allows simultaneous processing of different matrix data portions across multiple partitions, dramatically increasing processing throughput while maintaining a modular structure that manages complexity.
Solution Approach 2:
The media access circuitry is designed with universal functionality to perform both traditional memory access operations and tensor operations. This multi-functionality allows the same hardware structure to handle various computational tasks without requiring separate specialized processing units, thus increasing speed without proportionally increasing complexity.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
Technologies for providing high efficiency compute architecture on cross point memory for artificial intelligence operations include a memory that includes media access circuitry coupled to a memory media having a cross point architecture. The media access circuitry is to access matrix data from the memory media, including broadcasting matrix data associated with one partition of the memory media to multiple other partitions of the memory media. The media access circuitry is also to perform, with each of multiple compute logic units associated with different partitions of the memory media, a tensor operation on the matrix data and write, to the memory media, resultant data indicative of a result of the tensor operation.