Distributed Storage Resharding via Shard Header Accumulation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional distributed storage systems face performance issues and prolonged downtime during resharding due to the need to adjust the number of omap shards, requiring the system to be taken offline for metadata migration, which increases downtime as data volume grows.
Innovation Solution
A resharding method and system that accumulates shard statistical information into a header field, recreates new shard statistical information, and regroups operation logs, allowing resharding to occur while the system is online without migrating metadata, thus maintaining service availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the system uses traditional omap-based sharding with metadata migration during resharding, then the shard data can be redistributed according to the new number of shards, but the system must be taken offline for an extended period resulting in service interruption
Solution Approach 1:
The patent applies preliminary action by pre-calculating the new shard distribution plan and preparing the resharding configuration before actually executing the metadata migration. The system determines the target shard assignments for all objects in advance, so that when resharding begins, the actual data movement can proceed efficiently without intermediate decision-making delays, reducing overall downtime while maintaining service availability.
2Reliability
If the system performs consistency verification on large volumes of shard file data, then data integrity can be ensured, but the verification process takes a long time extending resharding duration
Solution Approach 1:
The patent applies partial action by performing consistency verification selectively rather than on all shard files. The system verifies consistency only for the subset of metadata that actually changes during resharding operations, such as objects being moved between shards or new shards being created. This selective verification approach maintains data reliability for critical changes while significantly reducing the overall verification time compared to checking every single shard file.
3Device complexity
If the system stores all metadata of a single bucket in the same shard file, then the storage structure is simple, but excessive shard file volume consumes large underlying storage performance and prolongs consistency verification
Solution Approach 1:
The patent applies segmentation by dividing the bucket metadata into multiple shard files distributed across different storage nodes. Each shard file contains a subset of the bucket's metadata rather than all metadata being stored in a single file. This segmentation improves storage performance by distributing the I/O load across multiple files and nodes, enables parallel consistency verification of different shard segments, and allows the system to handle larger volumes of metadata without degrading performance, while maintaining manageable file sizes for each individual shard.
Data Source
AI summary
The present application discloses a resharding method and system for a distributed storage system, and the method comprises: when performing resharding for the bucket, accumulating a statistical value in each of pieces of shard statistical information into a header field; deleting each of pieces of shard statistical information, and creating, according to the number of shards after resharding, several pieces of new shard statistical information, wherein, the number of the pieces of the new shard statistical information is equivalent to the number of shards after resharding, and the several pieces of new shard statistical information have one-to-one correspondence with the shards after resharding; and grouping operation logs in an operation log set according to the number of shards after resharding, such that the number of groups of the operation logs is consistent with the number of shards after resharding.