Partitioned Row Limiting for Hierarchical Query Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database query methods struggle with efficiently retrieving hierarchical data while avoiding duplicates and optimizing processor and memory usage, particularly in scenarios involving multi-vector queries and partitioned row limiting.
Innovation Solution
Implementing a hierarchical sorting approach with partitioned sorts and duplicate elimination on non-sort keys, utilizing multiple sorts to dynamically track and filter results based on hierarchical relationships, and optimizing memory usage by maintaining separate data structures for different tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a post-processing approach is used to retrieve all matching chunks and then filter results, then the query can be expressed in simple SQL, but processor and memory efficiency deteriorates because more results are retrieved than necessary
Solution Approach 1:
The patent segments the row limiting operation into multiple independent partition operations. Each partition handles a specific subset of rows based on partition keys, allowing the database to limit rows at the partition level rather than processing all rows globally. This segmentation enables efficient processing by dividing the large result set into manageable chunks that can be processed in parallel.
Solution Approach 2:
The patent performs preliminary sorting and partitioning operations before the final row limiting. By pre-sorting rows according to partition keys and order keys, and pre-dividing them into partitions, the system can efficiently apply row limits without needing to retrieve and post-process all matching rows. This preliminary action reduces the processing burden significantly.
2Reliability
If all matching paragraphs from multiple documents are retrieved, then complete search results are obtained, but memory footprint and processor cycles increase due to redundant data
Solution Approach 1:
The patent segments the search results by document ID into separate partitions. Each partition contains only the paragraphs from a specific document, allowing the system to limit and process each document's paragraphs independently. This segmentation prevents memory bloat by avoiding the need to hold all paragraphs from all documents in memory simultaneously.
Solution Approach 2:
The patent extracts and removes redundant data by eliminating duplicate paragraph representations across documents. Through partitioned processing, the system identifies and removes redundant data structures that would otherwise consume excessive memory, while preserving the complete search results through the partitioned approach.
3Measurement precision
If duplicate elimination is performed on non-sort keys after sorting, then accurate results are obtained, but processing time and complexity increase
Solution Approach 1:
The patent performs duplicate elimination as a preliminary action during the sorting process itself, rather than as a subsequent post-processing step. By integrating duplicate elimination into the sort operation, the system removes duplicates early when data is being organized, reducing the amount of data that needs to be processed later and thereby reducing overall processing time while maintaining accuracy.
Solution Approach 2:
The patent merges the sorting operation with the duplicate elimination operation into a single integrated process. By combining these two operations, the system avoids performing them sequentially, which would increase processing time. The merged operation achieves both sorting and duplicate removal simultaneously, improving efficiency while maintaining result accuracy.
Data Source
AI summary
A query is received for stored data items that have a plurality of attributes that include a first attribute and a second attribute that has a hierarchical relationship with the first attribute. A first sort of the stored data items is performed based on a first set of ordering keys that include the first attribute and the second attribute. A second sort of the stored data items is performed based on a second set of one or more ordering keys, the second set being a proper subset of the first set and including the second attribute as an ordering key. First pointers to the stored data items are inserted into a first sorted structure of the second sort of the stored data items. Second pointers to the stored data items are inserted into a second sorted structure of the second sort of the stored data items.