Processing-Near-Memory Storage for AI Query Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI query processing systems face challenges in efficiently handling increased query lengths and concurrent users due to limitations in memory bandwidth and capacity, leading to high system costs and inefficiencies.
Innovation Solution
The implementation of processing-near-memory (PNM) storage devices that process key values from data with transposed query values, determine probability distributions, and generate activation values, while also utilizing die-to-die communication to combine activation values from multiple PNM devices, thereby offloading computationally intensive tasks from GPU systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional GPU-based AI query processing systems are used, then computational power is sufficient, but memory bandwidth and capacity become limiting factors as query lengths and concurrent users increase
Solution Approach 1:
The patent segments the AI query processing system into two distinct parts: (1) GPU-based computational units that handle weight operations and matrix multiplications, and (2) PNM storage devices that handle data storage and retrieval operations. This segmentation allows each component to be optimized independently, with PNM devices providing high memory bandwidth and capacity while GPUs focus on computational tasks, thereby resolving the memory bandwidth bottleneck that limits query processing capacity in traditional unified GPU systems
Solution Approach 2:
The patent introduces PNM storage devices as intermediary components between the GPU computational units and the data storage layer. These PNM devices act as mediators that buffer and manage data flow, providing high-speed access to query and key-value data while offloading memory management tasks from the GPUs. This intermediary layer effectively decouples the computational power of GPUs from the memory bandwidth constraints, enabling the system to handle increased query lengths and concurrent users without proportionally increasing GPU resources
2Productivity
If more GPU systems are deployed to handle increased query lengths and concurrent users, then processing power increases, but system costs increase significantly
Solution Approach 1:
By segmenting the system into PNM storage devices and GPU computational units, the patent enables independent scaling of each component. Organizations can increase concurrent user handling capacity by adding PNM storage devices at lower cost, rather than deploying expensive GPU systems. The PNM devices handle the memory-intensive operations, allowing GPUs to be used more efficiently and reducing the total number of GPUs needed, thereby controlling system costs while maintaining productivity
Solution Approach 2:
The patent uses PNM storage devices to create copies and caches of frequently accessed data (such as key-value pairs and query data) close to the computational units. This copying mechanism reduces the need for repeated expensive GPU memory accesses and enables parallel processing of multiple queries without proportionally increasing GPU requirements, thus handling more concurrent users cost-effectively
3Speed
If data is frequently transferred between storage and processing units, then data access speed improves, but communication overhead increases
Solution Approach 1:
The patent merges the storage and processing functions into a unified architecture where PNM storage devices are closely coupled with GPU computational units through high-speed interconnects. This merging eliminates the need for frequent data transfers over long distances, as data can be accessed directly from PNM storage by the GPUs through optimized data paths. The unified architecture reduces communication overhead and energy loss while maintaining high data access speed, as the PNM devices provide persistent memory access without requiring constant data movement between separate storage and processing systems
Data Source
AI summary
Provided are systems, methods, and apparatuses for artificial intelligence query processing by processing-near-memory storage. In one or more examples, the systems, devices, and methods include receiving, at a first processing-near-memory (PNM) storage device, data and processing, at the first PNM storage device, first values from the data with transposed query values from the data. In one or more examples, the systems, devices, and methods include determining, at the first PNM storage device, a probability distribution of a result of the processing and generating, at the first PNM storage device, an activation value based on the probability distribution, the activation value indicating a correlation between units of text in a query associated with the data.