Time-Series Vector Embedding Delta Compression for Lower Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for storing and managing vector embeddings of time-series data are inefficient in terms of storage size, as they do not effectively utilize temporal adjacency to reduce file size.
Innovation Solution
The system generates delta encodings that describe differences between vector embeddings at adjacent time points, storing these encodings separately to reduce storage requirements, and uses these encodings to reconstruct vector embeddings for any time point in the series.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If vector embeddings are stored in traditional formats, then complete data is preserved, but storage size increases significantly
Solution Approach 1:
The patent segments vector embedding storage into two parts: (1) full vector embeddings at selected time points, and (2) delta encodings representing differences between adjacent time points. This segmentation allows the system to store only changes rather than complete vectors at every time point, significantly reducing storage size while maintaining the ability to reconstruct complete data when needed.
Solution Approach 2:
The patent transforms the storage representation from absolute vector values to relative delta encodings. By storing differences (parameter changes) between adjacent time points rather than absolute values, the system reduces storage requirements while preserving the ability to reconstruct original vectors through cumulative addition of deltas.
2Quantity of substance
If delta encodings are stored to reduce file size, then storage efficiency improves, but data reconstruction complexity increases
Solution Approach 1:
The patent performs preliminary organization of delta encodings in a structured format that facilitates efficient reconstruction. By pre-organizing delta encodings to correspond with their parent time points and maintaining an indexed structure, the system reduces the complexity of reconstruction operations, allowing for straightforward cumulative addition to recover original vectors.
Data Source
AI summary
A method of vector database compression includes receiving first time-series vector data comprising a first plurality of vectors representative of data corresponding to a first time point, receiving second time-series vector data comprising a second plurality of vectors representative of data corresponding to a second time point, generating a plurality of delta encodings, discarding the second time-series vector data after generating the plurality of delta encodings, and modifying database data to store the plurality of delta encodings. Each vector of the second plurality of vectors corresponds to one vector of the first plurality of vectors. The plurality of delta encodings is generated by, for all corresponding vectors of the first plurality of vectors and the second plurality of vectors, generating one delta encoding that describes differences between values of corresponding elements of the vector of the second plurality of vectors and the corresponding vector of the first plurality of vectors.