Distributed Storage Fragment Splitting for Availability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face reduced availability due to increased data volume and request pressure, where the need to split fragments for storage results in the target fragment becoming non-writable, reducing system availability.
Innovation Solution
A method where a control node sends a ready-to-split instruction to a master server, which then sends it to slave servers, allowing them to obtain and share data splitting points, enabling splitting-storage on the target fragment while maintaining its writability, thus improving system availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the target fragment is set to non-writable state during splitting, then data consistency is ensured, but system availability is reduced
Solution Approach 1:
The system performs preliminary actions by having slave servers calculate and send data splitting points to the master server before the actual splitting operation. The master server receives these splitting points in advance and uses them to perform the splitting operation, allowing the fragment to remain writable during the process while ensuring data consistency through pre-coordinated splitting points.
Solution Approach 2:
The master server acts as an intermediary between slave servers and the control node. It collects data splitting points from slave servers, coordinates the splitting operation, and manages the fragment state transitions. This intermediary role allows for coordinated splitting that maintains data consistency while minimizing disruption to system availability.
2Reliability
If the target fragment is set to non-writable state during splitting, then data consistency across servers is ensured, but write operations are denied
Solution Approach 1:
Slave servers perform preliminary calculations to determine data splitting points and send these to the master server before the splitting operation begins. This preliminary action allows the master server to have the splitting information ready, enabling it to perform the split while the fragment remains writable, thus maintaining write operation capacity.
Solution Approach 2:
The system dynamically manages the fragment state, transitioning it between writable and non-writable states based on the progress of the splitting operation. Rather than keeping the fragment permanently non-writable, the state is dynamically adjusted to allow write operations during safe periods while ensuring consistency during critical splitting moments.
3Reliability
If multiple servers store copies of the same fragment, then fault tolerance is improved, but splitting complexity increases
Solution Approach 1:
The system segments the splitting coordination task by assigning specific roles to different servers: slave servers calculate and send splitting points, while the master server coordinates the actual splitting operation. This segmentation of responsibilities simplifies the overall complexity by dividing it into manageable, specialized tasks distributed across servers.
Solution Approach 2:
The master server serves as an intermediary that centralizes the coordination of splitting operations across multiple servers. It collects splitting points from slave servers, manages the fragment state transitions, and coordinates the actual splitting execution. This intermediary approach simplifies the complexity by providing a single point of coordination rather than requiring all servers to coordinate directly with each other.
Data Source
AI summary
In one example, a method includes: receiving a ready-to-split instruction for the to-be-split fragment; sending the ready-to-split instruction to slave servers, so that each of the slave servers obtains data splitting points of the to-be-split fragment, and sends the data splitting points to the master server; when the number of received data splitting points exceeds a first threshold number, sending data splitting points to the control node, so that the control node sends a splitting-storage instruction to the master server; sending the splitting-storage instruction to each of the slave servers, so that each of the slave servers performs splitting-storage on the to-be-split fragment according to the data splitting points, and sends a first message to the master server to indicate completion of the splitting-storage; when the number of received first messages exceeds a second threshold number, performing splitting-storage on the to-be-split fragment according to the data splitting points.