Shard Splitting via Hard Linking and Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Subdividing a shard in distributed data systems, such as those used in Elasticsearch, is a fragile and time-consuming process, often requiring reindexing which can take hours or days and is prone to corruption due to changes during the process, and poses technical challenges in managing large datasets that exceed hardware limits.
Innovation Solution
A method for shard splitting involves marking a source index as read-only, creating a target index with more shards, hashing identifiers to assign documents to new shards, hard linking files, and deleting documents from the wrong shards, allowing for quick and fault-tolerant shard splitting without reindexing, thereby minimizing downtime and risk of data corruption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional reindexing is used to subdivide shards, then data can be redistributed, but the process is time-consuming (hours or days) and prone to corruption
Solution Approach 1:
The patent creates a copy of the source shard data structure (including segment files and term dictionaries) to form the target shard. This copying approach allows the target shard to be initialized quickly without performing time-consuming reindexing operations, while maintaining data consistency through the copy operation.
Solution Approach 2:
The patent performs preliminary actions by pre-calculating the target shard structure, segment assignments, and term dictionary mappings before actual data redistribution. This includes creating the target index metadata, organizing segment files, and preparing term dictionaries in advance, which significantly reduces the overall shard splitting time.
2Ease of operation
If reindexing is performed to split shards, then data redistribution can occur, but the process is fragile and prone to corruption due to changes during processing
Solution Approach 1:
The patent segments the shard splitting process into distinct, independent phases: creating target index metadata, copying segment files, copying term dictionaries, and finalizing the target shard. Each phase is self-contained and can be validated independently, reducing the risk of corruption compared to monolithic reindexing operations.
Solution Approach 2:
The target shard structure is designed to be self-consistent with embedded metadata that automatically tracks segment assignments and term dictionary mappings. This self-describing structure allows the system to validate its own integrity without requiring complex external verification, reducing corruption risks.
3Quantity of substance
If traditional methods are used to manage large datasets exceeding hardware limits, then data can be stored, but shard management becomes complex and time-consuming
Solution Approach 1:
The patent introduces a new dimension of organization by creating target shards with a refined hierarchical structure that includes detailed segment-to-shard mapping metadata and term dictionary indexes. This additional structural dimension enables more efficient data location and retrieval, reducing the complexity of managing large datasets across multiple shards.
Data Source
AI summary
Methods and systems for shard splitting are provided. Exemplary methods include: marking a source index as read only, the source index comprising a source shard, the source shard comprising a source reference; creating a target index, the target index comprising target shards, each target shard of the target shards comprising a target reference of target references; copying the source reference, the copying producing the target references; hashing identifiers in the source reference, each identifier being associated with a document of a plurality of documents of the source shard, the hashing assigning each document of the plurality of documents to a target shard of the target shards, the plurality of documents being stored in a file associated with the source reference; hard linking the file into the target references; marking the target index as read-write; and deleting the source index.