Embedding Management Unit for SSD-Based Vector Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural information retrieval systems require significant main memory to store document embedding vectors, leading to high memory usage and costs, especially when dealing with large datasets, as a 3 GB text document can generate 150 GB of document embeddings, multiplying to substantial memory requirements with millions of documents.
Innovation Solution
Implementing a system that uses a Solid State Drive (SSD) to store document embeddings, with an Embedding Management Unit (EMU) to manage the storage and transfer of these vectors between SSD, main memory, and a local accelerator, allowing for efficient caching and prefetching to reduce memory usage, and utilizing a multi-level cache architecture to optimize query processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If document embedding vectors are stored in main memory during query processing, then query processing speed is improved, but main memory requirements increase significantly
Solution Approach 1:
The system segments the storage of document embedding vectors across multiple memory levels: a first memory (e.g., SSD) for bulk storage and a second memory (e.g., RAM) for active caching. This segmentation allows the system to maintain fast query processing for recently accessed documents while storing the majority of documents in cheaper, lower-speed memory, thus resolving the contradiction between query speed and memory requirements.
Solution Approach 2:
The patent introduces a temporal dimension to memory management through prefetching and caching mechanisms. By predicting which documents will be needed soon and loading them into the second memory in advance, the system creates a time-based layer that reduces the immediate memory burden during query processing while maintaining performance.
2Quantity of substance
If a multi-level cache architecture with SSD and local accelerator is implemented, then main memory requirements are reduced, but system complexity increases
Solution Approach 1:
The patent introduces an intermediary management layer that coordinates between the first memory (SSD), second memory (RAM), and local accelerator. This intermediary layer handles the complexity of data migration, caching policies, and prefetching operations, allowing the system to reduce main memory requirements while managing the increased architectural complexity through a dedicated control mechanism.
3Productivity
If prefetching and caching mechanisms are implemented, then query processing efficiency is improved, but memory transfer overhead increases
Solution Approach 1:
The system performs preliminary actions by prefetching documents into the second memory before they are actually needed for query processing. This is achieved by analyzing query patterns and predicting future document access needs, allowing the system to load data in advance and reduce the frequency of memory transfers during actual query execution, thus improving efficiency while managing transfer overhead.
Data Source
AI summary
A system is disclosed. A processor may include a local memory. A memory may be connected to the processor. A cache-coherent interconnect storage device may also be connected to the processor. An embedded management unit (EMU) may be configured to manage the storage of a document embedding vector in the local memory, the memory, or the cache-coherent interconnect storage device.