Recursive Document Partitioning for Low-Merge Vector Database Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing vector database systems require complex and inefficient methods for updating or deleting documents, leading to high merging rates and poor parallelism during re-computation, especially when using external mappings and chunk-based approaches.
Innovation Solution
Implement a computer-based method involving sorting documents by frequency updates, creating recursive partitions using Monte Carlo or simulated annealing, computing the cost of these partitions, and merging based on the partition with the smallest cost to minimize merging rates and improve parallelism.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If external mapping and chunk-based approaches are used for document updates, then document tracking capability is improved, but merging rate increases and parallelism deteriorates
Solution Approach 1:
The patent divides the document update process into independent partition operations. Instead of processing all documents globally, the system segments documents into partitions and processes each partition independently, enabling parallel execution and reducing the overall merging rate while maintaining document tracking capability through partition-level metadata.
Solution Approach 2:
The patent introduces a new dimension of parallelism by processing multiple document partitions simultaneously across different computing resources. This dimensional transformation from sequential global processing to parallel partitioned processing reduces the merging rate while preserving document tracking through distributed metadata management.
2Reliability
If external mapping and chunk-based approaches are used for document updates, then document tracking capability is improved, but parallelism deteriorates
Solution Approach 1:
The patent segments the document set into independent partitions that can be processed in parallel. Each partition maintains its own metadata for document tracking, enabling simultaneous processing across multiple computational units while preserving the ability to track individual documents within their respective partitions.
Solution Approach 2:
The patent performs preliminary partitioning of documents before update operations. By pre-organizing documents into independent partitions with their own metadata structures, the system enables immediate parallel processing without compromising document tracking capability, thus improving adaptability for parallel execution.
3Adaptability or versatility
If complex update methods are used in vector databases, then document management capability is improved, but system complexity increases
Solution Approach 1:
The patent simplifies the overall system by segmenting the document management process into independent partition operations. Each partition can be managed separately with simpler logic, reducing the complexity of the overall system while maintaining versatile document management capabilities through the coordinated operation of multiple partitions.
Solution Approach 2:
The patent enables each partition to manage its own documents and metadata independently through self-service operations. This autonomy reduces the need for complex centralized coordination mechanisms, thereby reducing system complexity while preserving comprehensive document management capability across all partitions.
Data Source
AI summary
Embodiments sort a plurality of documents in an increasing order of frequency updates, create multiple recursive partitions of the plurality of sorted documents, compute a cost of the multiple recursive partitions, choose a partition with a smallest cost from computed costs of the multiple recursive partitions, and merge the plurality of documents based on the partition with the smallest cost.