Distributed Storage Range Subdivision for Parallel Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage systems, the fixed number of cluster nodes can lead to inefficient data processing throughput due to either serial execution of hot ranges or increased routing overhead when ranges are partitioned based on primary or non-primary keys, affecting read/write efficiency and system performance.
Innovation Solution
The method involves partitioning a range into subranges on a cluster node, establishing a log queue for each subrange, and managing data operations and synchronization within these subranges to enable parallel processing, thereby improving throughput without increasing client routing overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a node maintains a small quantity of ranges, then data read/write efficiency is improved, but routing overhead increases for clients
Solution Approach 1:
The patent segments a range into multiple subranges, with each subrange independently managed by a cluster node. This allows finer-grained parallel processing within a range while maintaining the original range interface for clients, thus improving data read/write efficiency without increasing routing overhead.
2Reliability
If multiple threads compete for permission on the same range, then serial execution occurs, but system reliability is maintained
Solution Approach 1:
By dividing a range into multiple subranges, the patent enables multiple threads to operate on different subranges simultaneously without conflicting for permission on the same range. This segmentation eliminates serial execution bottlenecks while maintaining data consistency within each subrange, thus improving throughput while preserving reliability.
Solution Approach 2:
The patent introduces a new dimension of granularity by creating subranges within ranges. This dimensional transformation allows parallelism at the subrange level while maintaining the original range structure for client interactions, effectively resolving the throughput-reliability contradiction.
3Productivity
If a node maintains a large quantity of ranges, then data processing throughput is improved, but routing overhead increases
Solution Approach 1:
The patent segments ranges into subranges that are maintained by cluster nodes, while clients continue to use the original range structure for routing. This allows the system to achieve fine-grained parallel processing for high throughput without exposing the segmentation complexity to clients, thus improving throughput without increasing routing overhead.
Data Source
AI summary
Disclosed are a distributed storage system, a cluster node, and a range management method thereof, including: partitioning, by the cluster node according to a configuration parameter, a range corresponding to a first routing table entry in a local routing table into at least two subranges, where the first routing table entry refers to a routing table entry in which information that indicates the cluster node is recorded in replica information; separately establishing, by the cluster node, a log queue for each of the subranges; determining, by the cluster node, a corresponding subrange according to a key field of a data operation request from a client; executing a corresponding data read/write operation according to the determined subrange; and updating, according to the data read/write operation, a log queue corresponding to the determined subrange.