Embedded Database Vector Search With Disk-Backed Vectors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional vector search techniques in embedded databases face challenges with high memory consumption and limited flexibility in updating indexes, particularly in resource-constrained environments, leading to poor performance and system crashes.
Innovation Solution
A system that stores centroids in memory and vectors in persistent storage, using an ordered index like a B-tree to efficiently map centroids to vectors, reducing memory usage and enabling fast lookup, while processing queries based on a subset of vectors loaded in memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If vectors are stored in memory for vector search, then search speed is improved, but memory consumption increases significantly
Solution Approach 1:
The patent segments the vector data into two parts: centroids (stored in memory) and full vectors (stored on disk). This segmentation allows the system to maintain fast search capability through in-memory centroids while avoiding the memory consumption problem of storing all vectors in memory. The B-tree index further segments the data structure to enable efficient disk-based storage with fast lookup.
Solution Approach 2:
The patent introduces a hierarchical dimension to the storage architecture, moving from a flat in-memory structure to a multi-level structure with centroids in memory and full vectors on disk. The B-tree index adds another dimensional layer, organizing vectors in a tree structure that enables logarithmic-time lookup from centroids to full vectors, effectively trading space for time in a controlled manner.
2Speed
If conventional vector indexing stores both centroids and vectors in memory, then lookup speed is improved, but device reliability deteriorates due to memory constraints
Solution Approach 1:
The patent extracts the full vectors from memory and stores them on disk, keeping only the essential centroids in memory. This extraction resolves the memory constraint problem that causes system crashes while maintaining search functionality through the centroid-based indexing and B-tree structure that enables fast disk-based retrieval.
3Quantity of substance
If vectors are stored on persistent storage instead of memory, then memory efficiency is improved, but access speed deteriorates
Solution Approach 1:
The patent performs preliminary organization of vectors into a B-tree structure on disk, with centroids pre-computed and stored in memory. This preliminary action enables fast access during query time by allowing the system to quickly navigate the pre-organized B-tree structure and retrieve only the necessary vectors from disk, rather than scanning all vectors linearly.
Solution Approach 2:
The B-tree index acts as an intermediary between the in-memory centroids and the disk-stored vectors. It provides a efficient lookup mechanism that minimizes disk seeks by organizing vectors in a balanced tree structure, where the centroid serves as the root and vectors are organized in subtrees, reducing the access path from O(N) to O(log N).
Data Source
AI summary
A system stores a vector index based on vector data stored in a database and processes queries based on vector data using the vector index. The system stores a plurality of records in a database. The system receives a plurality of vectors, each vector associated with a record. The system stores, in a persistent storage, a vector index including tuples. Each tuple stores a vector, and an identifier of a record associated with the vector. The system receives a database query requesting a result set associated with a target record. The system identifies a target vector associated with the target record. The system accesses the vector index to retrieve a subset of vectors based on a distance from the target vector. The subset of vectors is loaded in memory from the persistent storage. The system determines a result set based on the subset of vectors.