Embedding Vector Grouping for Storage Read Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer storage systems face increased read latency when processing machine learning workloads, particularly in deep-learning recommender-model (DLRM) inference workloads, due to the large number of embedding vectors that need to be retrieved from storage.
Innovation Solution
The proposed solution involves grouping embedding vectors in a storage device using a classification model that determines access frequencies, allowing frequently accessed vectors to be stored in cache memory while less frequently accessed vectors are stored in persistent storage, leveraging a cache coherent protocol such as Compute Express Link (CXL).
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If all embedding vectors are stored in persistent storage, then storage capacity is maximized, but read latency increases significantly
Solution Approach 1:
The patent segments embedding vectors into two distinct storage locations based on access frequency: frequently accessed vectors are placed in cache memory while less frequently accessed vectors are stored in persistent storage. This segmentation allows the system to maintain large storage capacity while reducing read latency for commonly accessed data by serving it from the faster cache memory.
Solution Approach 2:
The patent applies local quality by differentiating storage characteristics for different subsets of embedding vectors. High-frequency access patterns receive preferential treatment with cached storage, while low-frequency patterns use persistent storage. This creates non-uniform storage quality matched to access requirements, optimizing the capacity-latency tradeoff.
2Loss of time
If embedding vectors are cached to reduce read latency, then read performance improves, but storage requirements for cache memory increase
Solution Approach 1:
The patent implements dynamic cache management using a classification model that continuously monitors access patterns and adjusts which embedding vectors are cached. The system dynamically adapts cache contents based on real-time access frequency, allowing efficient use of limited cache capacity by prioritizing the most valuable (frequently accessed) vectors.
Solution Approach 2:
The patent changes the parameter of access frequency classification to determine cache eligibility. By thresholding access frequency and using machine learning models to predict patterns, the system dynamically determines which vectors qualify for caching, optimizing the balance between cache utilization and memory capacity requirements.
3Productivity
If a classification model is used to determine access frequency, then storage optimization is achieved, but system complexity increases
Solution Approach 1:
The patent implements self-service by enabling the storage system to autonomously monitor, classify, and manage embedding vector access patterns without external intervention. The classification model automatically learns from access data and makes decisions about caching, freeing the host system from complex manual management while achieving optimized storage efficiency.
Solution Approach 2:
The patent incorporates feedback mechanisms where access frequency data is continuously collected and fed back into the classification model. This feedback loop allows the system to refine its predictions and improve caching decisions over time, achieving high storage efficiency through adaptive learning rather than static rules.
Data Source
AI summary
A method of grouping embedding vectors in a storage device includes receiving, at a storage device, an inference query request including a first access index associated with a first embedding vector, determining, with a classification model, a first classification of the first access index, based on the first classification, searching for the first embedding vector in a cache of the storage device, receiving, at the storage device, a second access index associated with a second embedding vector, determining, with the classification model, a second classification of the second access index, and based on the second classification, searching for the second embedding vector in a persistent storage of the storage device.