Vector Database Versioning for Seamless Model Evolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing vector processing workloads face challenges in providing fast response times and efficient management of vector databases, especially in handling high volumes of vectors generated by machine-learned models, which require specialized indexing and real-time updates, while existing solutions are often CPU-intensive and lack flexibility in handling different vector dimensions and formats.
Innovation Solution
A vector database management system (VDBMS) with a new VECTOR datatype that supports flexible dimension counts and formats, allows versioning of vectors within a single object, and includes optimized indexing and retrieval mechanisms to manage multiple vector versions efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If vectors are versioned to support model evolution, then adaptability is improved, but device complexity increases
Solution Approach 1:
The patent implements versioning by nesting version metadata within the vector object structure itself. Each vector object contains a version number field that is embedded within the object, allowing multiple versions to be stored in a hierarchical manner without requiring separate versioning tables or complex external management mechanisms.
Solution Approach 2:
The vector object structure is designed to be universal and multi-functional, simultaneously storing vector data, version information, and metadata in a single integrated structure. This allows the same object type to handle both the vector itself and its versioning requirements, eliminating the need for separate specialized structures.
2Quantity of substance
If multiple vector versions are stored in a single object, then storage efficiency is improved, but retrieval complexity increases
Solution Approach 1:
The patent segments version retrieval by introducing a dedicated version number field within the vector object that can be independently queried. This allows the system to efficiently retrieve specific versions by filtering on the version number field rather than searching through all vector data, simplifying the retrieval process despite storing multiple versions in a single object.
3Productivity
If real-time vector updates are supported, then productivity is improved, but loss of time increases
Solution Approach 1:
The patent implements dynamic versioning where vector objects can be updated in real-time by modifying the version number field and associated data. The system supports concurrent reads and writes by managing version numbers dynamically, allowing updates to propagate immediately without requiring complex transaction management or locking mechanisms that would cause significant delays.
Data Source
AI summary
Techniques for creating and maintaining versions of vectors in a database statement are provided. In one technique, a first version of a vector is stored in a vector object. A second version, of the vector, that is different than the first version and that is not stored in the vector object is identified. In response to receiving an instruction to store the second version in the vector object, the vector object is identified, the vector object is updated to include the second version in addition to the first version, and a value that indicates a location, within the vector object, of the first version or of the second version is inserted into a next version reference field of the vector object.