Distributed Storage Data Distribution via Node Group Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage systems using erasure coding, concurrent read-write operations by multiple data distribution apparatuses lead to frequent lock contending and switching, resulting in significant read-write delays due to the need for distributed locks to manage conflicts, which fails to meet actual application requirements.
Innovation Solution
The data distribution method involves dividing data into logical partitions and distributing each partition to a primary node within a node group, where erasure coding is performed by the primary node, eliminating the need for distributed locks and reducing lock switching, thereby improving system performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If distributed lock is used to solve conflict between multiple data distribution apparatuses, then data consistency is maintained, but read-write delay increases due to frequent lock contending and lock switching
Solution Approach 1:
The patent segments the data storage system into multiple node groups, where each node group independently handles specific EC stripes. This segmentation allows different data distribution apparatuses to operate on different node groups simultaneously without requiring global distributed locks, thereby maintaining data consistency within each segment while eliminating lock contending between segments.
Solution Approach 2:
The patent introduces node groups as intermediary layers between data distribution apparatuses and storage nodes. Each node group acts as a mediator that manages its own EC stripes independently, preventing direct conflict between multiple data distribution apparatuses and eliminating the need for distributed lock management.
2Productivity
If multiple data distribution apparatuses concurrently read or write data in a same EC stripe, then system throughput is improved, but lock contending occurs requiring distributed locks
Solution Approach 1:
EC stripes are segmented and assigned to specific node groups, allowing multiple data distribution apparatuses to concurrently operate on different node groups without interference. This enables high system throughput while eliminating the need for complex distributed lock management to handle concurrent access.
3Ease of operation
If data distribution apparatus performs erasure coding, then data distribution control is centralized, but conflict with other apparatuses increases requiring distributed locks
Solution Approach 1:
The patent extracts the erasure coding function from the data distribution apparatus and relocates it to the node group level. This allows data distribution apparatuses to focus solely on distribution control while node groups independently perform erasure coding on their assigned EC stripes, eliminating conflicts between apparatuses performing coding operations.
Data Source
Figure 1(a)
Figure 1(b)
Figure 2~3
AI summary
Embodiments of the present invention disclose a data distribution method for improving performance of a distributed storage system. The method in the embodiments of the present invention includes: receiving, by a data distribution apparatus, a storage instruction of a user, dividing to-be-stored data that the storage instruction instructs to store, into P data segments, determining a storage node group corresponding to each data segment, and finally distributing the data segment to a primary node in the corresponding storage node group. The embodiments of the present invention further provide a related data storage method, a related apparatus, and a system.