Sparse Vector Storage Formats for Faster Database Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing vector databases struggle with efficient storage and processing of sparse vectors, which have a large number of dimensions but few non-zero values, leading to inefficient indexing and searching, especially in high-volume generative AI applications.
Innovation Solution
A vector database system that supports sparse vectors by representing them as two arrays of values and selecting an appropriate storage format based on criteria such as distance functions and non-zero value distribution, with flexible dimension counts and formats, and allows for versioning and caching metadata to accelerate operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If sparse vectors are stored using traditional dense vector storage methods, then the storage structure is simple, but the storage efficiency is low and computational overhead is high
Solution Approach 1:
The patent segments the storage structure by separating non-zero elements from zero elements in sparse vectors. It uses two arrays: one storing only non-zero values and another storing their corresponding positions. This segmentation eliminates wasted storage space while maintaining simple access patterns, directly resolving the contradiction between storage efficiency and structure complexity.
Solution Approach 2:
The patent changes the storage parameter from dense format (storing all dimensions including zeros) to sparse format (storing only non-zero elements with position metadata). This parameter change transforms the storage efficiency from O(n) to O(k) where k is the number of non-zero elements, while the added position array keeps the structure manageable.
2Speed
If approximate indexing techniques are used for vector embeddings, then indexing speed is improved, but the precision of similarity search decreases
Solution Approach 1:
The patent applies local quality by using different indexing strategies for different parts of the sparse vector data. It uses position-based indexing for exact position matching and value-based indexing for similarity computation. This localized approach to different data aspects maintains both speed and precision by optimizing each aspect appropriately.
3Adaptability or versatility
If the vector database schema is fixed, then data consistency is maintained, but the system cannot accommodate evolving embedding models with different dimension counts
Solution Approach 1:
The patent implements dynamics by making the vector dimension parameter mutable and configurable. The schema allows dimension counts to be updated without migration, and the storage structure adapts dynamically to different dimension sizes. This dynamic approach maintains data consistency through version control while accommodating evolving models with different dimensions.
Solution Approach 2:
The patent creates a universal storage structure that can handle vectors of any dimension count. The sparse vector format with position arrays works regardless of whether the vector has 10 dimensions or 10,000 dimensions, making the system multi-functional and adaptable to different embedding models without requiring schema changes.
4Measurement precision
If all vector dimensions are processed during similarity search, then complete accuracy is achieved, but the computational time increases significantly for high-dimensional sparse vectors
Solution Approach 1:
The patent extracts and processes only the non-zero elements during similarity search operations. By taking out the relevant information (non-zero values and their positions) and ignoring the zero elements entirely, it achieves complete search accuracy while reducing computational time from O(n) to O(k) where k is the number of non-zero elements.
Data Source
AI summary
Techniques for generating and managing sparse vector representations in a database system are provided. In one technique, an embedding that was generated by an embedding model is accessed. Based on one or more characteristics associated with the embedding, a particular storage format is selected from among multiple storage formats in which to store the embedding. A sparse vector representation is generated based on the embedding and the particular storage format. The sparse vector representation is stored. The sparse vector representation may be stored in the same VECTOR type column that stores sparse vector representations that are in different storage formats and/or dense vector representations.