Dynamic Rolling Block Partitioning for Distributed Storage Growth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed storage systems face challenges in dynamically managing partitions to handle massive data, leading to complex data processing and increased maintenance costs due to frequent splitting and merging of partitions.
Innovation Solution
A data partitioning method where a master node in a distributed storage system continuously creates and splits rolling blocks into partitions, establishing a mapping relationship between partitions and storage nodes, allowing for dynamic partition adjustment without the need for frequent splitting and merging.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If hash-based partitioning or value range-based partitioning is used in distributed storage systems, then data can be stored across multiple storage nodes, but frequent splitting and merging of partitions is required when dealing with massive data, leading to complicated data processing and increased maintenance costs
Solution Approach 1:
The patent segments the distributed storage system into multiple independent rolling blocks, where each rolling block contains multiple partitions. This segmentation allows the system to handle massive data by adding new rolling blocks rather than continuously splitting existing partitions, thereby reducing partition management complexity while maintaining adaptability to data growth.
Solution Approach 2:
The patent implements dynamic partition management through the rolling block mechanism, where partitions are dynamically allocated within rolling blocks based on data writing needs. When a rolling block is full, a new rolling block is created rather than splitting existing partitions, providing dynamic adaptability to data growth without the complexity of frequent partition splitting and merging operations.
2Productivity
If partitions are frequently split and merged to handle data growth, then the system can adapt to increasing data volume, but maintenance costs and performance loss increase
Solution Approach 1:
The patent performs preliminary actions by pre-allocating multiple partitions within each rolling block before data writing begins. This allows the system to accommodate data growth by switching to new rolling blocks rather than performing time-consuming split and merge operations, thereby maintaining high productivity while reducing maintenance time.
Solution Approach 2:
The patent implements a rolling block mechanism where full rolling blocks can be discarded or archived while new rolling blocks are created for continued data writing. This approach avoids the time-consuming process of splitting and merging partitions, as the system can simply move to new rolling blocks, thereby maintaining productivity while minimizing maintenance time.
Data Source
AI summary
A data partitioning and processing method having the following steps: receiving a partitioned storage capacity sent by each storage node in a distributed storage system; wherein the partitioned storage capacity is a storage capacity of each partition included in the storage node; determining a storage capacity of an i-th rolling block according to the partitioned storage capacity; determining whether the storage capacity of the i-th rolling block meets a preset condition according to the storage capacity of the i-th rolling block; creating an (i+1)-th rolling block in response to the fact that the storage capacity of the i-th rolling block meets a preset condition; splitting the (i+1)-th rolling block into N partitions on average; and establishing a mapping relationship between N partitions in the (i+1)-th rolling block and storage nodes in the distributed storage system, so that the storage node writes data into the partitions according to the mapping relationship.