Distributed Storage Disk Selection via Dynamic Load and Capacity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large-scale online distributed storage systems, conventional disk selection strategies using consistency hash algorithms often lead to disks becoming fully written and experiencing hotspot access, resulting in write-in failures and extended access times when data is concentrated in specific partitions.
Innovation Solution
A two-stage disk selection method that involves selecting a target replica server based on remaining capacity and disk load, followed by choosing a write-in disk within that server using a probability determination process that considers capacity and load weights, to prevent disk overload and hotspot access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If consistency hash algorithm is used for disk selection, then data distribution balance is improved, but disk overload and hotspot access occur when data is concentrated in same partition
Solution Approach 1:
The patent changes the selection parameters from static hash-based partitioning to dynamic parameters including remaining capacity ratio and load factor. The system calculates real-time disk status parameters (remaining capacity, I/O load) and uses these changing parameters to select target disks, avoiding both full disks and hotspots
Solution Approach 2:
The patent transforms the static disk selection strategy into a dynamic one by continuously monitoring disk status and adjusting selection criteria. The system periodically updates remaining capacity and load information, making the selection process adaptive to changing system conditions rather than fixed based on initial hashing
2Reliability
If data is written to disks with high remaining capacity, then disk overload is prevented, but write-in time increases due to intensive traffic on selected disks
Solution Approach 1:
The patent introduces a composite selection criterion that changes from considering only remaining capacity to considering both remaining capacity ratio and load factor. This dual-parameter approach dynamically adjusts target disk selection to balance capacity utilization with current traffic conditions, reducing write-in time by avoiding overloaded disks
Solution Approach 2:
The system implements feedback by continuously monitoring disk load and using this information to adjust subsequent write-in decisions. The load factor provides real-time feedback about disk performance, allowing the system to adapt its selection strategy and avoid disks that are becoming bottlenecks
Data Source
AI summary
Embodiments of the disclosure provide a data write-in method and apparatus. The data write-in method includes: selecting a target replica server from a plurality of replica servers managed by a metadata server according to remaining capacity of the plurality of replica servers; selecting a write-in disk in the target replica server according to remaining capacity and load of disks managed by the target replica server; and storing write-in data into the write-in disk through the target replica server.