High-Access-Rate Data Transfer for Storage Node Congestion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems experience congestion at storage nodes due to uneven distribution of high-access-rate data objects, leading to processing delays and network bottlenecks that existing network-level and storage-node-level techniques fail to address effectively.
Innovation Solution
A computing system detects congestion conditions at storage nodes, identifies high-access-rate data objects, and relocates them to different storage nodes via computed transfer paths, using a scheduler and controller to manage data redistribution and offload high-access-rate objects, thereby balancing traffic and reducing congestion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data objects are stored at storage nodes in a distributed storage system, then data storage capacity is improved, but congestion occurs at storage nodes due to uneven distribution of high-access-rate data objects
Solution Approach 1:
The system dynamically identifies high-access-rate data objects and relocates them to different storage nodes based on access patterns. This creates non-uniform data distribution where frequently accessed objects are strategically positioned to minimize congestion, while less frequently accessed objects remain distributed across the network. The local quality of data placement is optimized based on access characteristics rather than uniform distribution.
Solution Approach 2:
The patent implements dynamic data object relocation by continuously monitoring access rates and transferring data objects between storage nodes in response to changing access patterns. The system adapts the distribution of data objects over time, transferring high-access-rate objects from congested nodes to less congested nodes, thereby dynamically balancing the load across the storage network.
2Adaptability or versatility
If multiple storage nodes are networked together to form a distributed storage system, then data accessibility is improved, but network congestion and processing delays occur
Solution Approach 1:
The system performs preliminary identification of high-access-rate data objects using access rate data and derivatives before congestion occurs. By proactively transferring these objects to different storage nodes in advance, the system prevents future congestion and reduces processing delays. The scheduler computes transfer paths and initiates transfers before the congestion fully manifests, thereby maintaining data accessibility while avoiding delays.
3Reliability
If existing network-level and storage-node-level techniques are used to manage data traffic, then basic data transfer is maintained, but congestion at storage nodes due to high-access-rate data objects is not effectively addressed
Solution Approach 1:
The system implements feedback mechanisms by continuously monitoring access rate data and its derivatives to identify changes in data object access patterns. This feedback drives automatic transfer decisions, where the scheduler detects when a data object becomes high-access-rate and initiates relocation to prevent congestion. The feedback loop ensures that data transfer reliability is maintained while optimizing processing throughput through adaptive load balancing.
Data Source
AI summary
A computing system including one or more processing devices configured to detect a congestion condition occurring at a first storage node located in a storage network of a distributed storage system. The one or more processing devices are further configured to obtain respective first access rate data for a first plurality of data objects stored at the first storage node. Based at least in part on the first access rate data, the one or more processing devices are further configured to flag a first data object as a high-access-rate data object. The one or more processing devices are further configured to compute a transfer path between the first storage node and a second storage node in the storage network. The one or more processing devices are further configured to transfer the high-access-rate data object from the first storage node to the second storage node along the transfer path.