Document Index Partitioning via Segment Splitting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional document index management systems face performance deterioration as data increases, and the number of partitions cannot be changed without rehashing all documents, making it inefficient to add new partitions.
Innovation Solution
A method and device for managing document indexes by determining an original segment, partitioning it into two portions, where one portion updates the original segment and the other becomes a new target segment, with index rebuilding only for the new segment, allowing for efficient addition of new partitions without reindexing all documents.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the number of partitions is increased to improve search performance and distribute data better, then system scalability and query performance are improved, but all documents must be rehashed and re-indexed, causing significant loss of time and increased complexity
Solution Approach 1:
The patent segments the partition modification process by identifying and extracting only the affected segment (segment B) that needs to be split, rather than reprocessing all segments. When adding a new partition, only the segment whose hash range overlaps with the new partition's range is divided, while other segments remain unchanged.
Solution Approach 2:
The patent extracts and processes only the necessary portion (segment B) that needs to be modified when adding a new partition. The affected segment is isolated and split into sub-segments, while the rest of the partition structure remains intact and requires no reindexing.
2Adaptability or versatility
If traditional rehashing is performed to add new partitions, then data distribution is optimized, but system performance deteriorates dramatically as data increases due to the overhead of reindexing all documents
Solution Approach 1:
The patent applies partial action by performing reindexing only on the affected segment (segment B) that needs to be split, rather than reindexing all segments in the partition. This partial reindexing maintains data distribution flexibility while minimizing the performance overhead associated with full rehashing operations.
3Device complexity
If the partition structure is made fixed to simplify management, then device complexity is reduced, but the system cannot adapt to changing data volumes and query requirements
Solution Approach 1:
The patent introduces dynamics into the partition structure by enabling segments to be split into sub-segments when needed. The segment tree structure allows for dynamic expansion where segments can be divided into multiple sub-segments (e.g., segment B divided into B1 and B2) based on query patterns and data distribution requirements, while maintaining a relatively simple base structure.
Data Source
AI summary
Embodiments of the present disclosure provide a method and device for managing document indexes. The method includes in response to a request for adding a target segment to a document set including a plurality of segments, determining an original segment associated with the target segment from the plurality of segments, partitioning the original segment into a first portion and a second portion, the first portion being used to update the original segment, the second portion being used as the target segment, and rebuilding indexes for documents in the target segment.