Dynamic Data Distribution via Resource Distribution Factor
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data storage systems face challenges in balancing performance, reliability, and cost, particularly in reducing the probability of data loss due to hardware or software failures, where traditional data distribution algorithms often result in high replication factors leading to increased hardware and operating costs.
Innovation Solution
A method and system that utilize a distribution engine to modify data distribution algorithms by calculating a resource distribution factor (RDF) to limit the number of nodes sharing data, mapping each storage resource to a candidate resource set, and selecting a final resource set based on this RDF, thereby reducing the probability of data loss and rebalancing requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional data distribution algorithms are used to map data to storage nodes, then data redundancy and reliability are improved, but hardware and operating costs increase due to high replication factors
Solution Approach 1:
The patent changes the parameter of data distribution by introducing a resource distribution factor (RDF) that limits the number of candidate nodes, thereby optimizing the replication factor dynamically rather than using fixed high replication to achieve reliability
Solution Approach 2:
The system dynamically adjusts data distribution based on current system state, failure scenarios, and load conditions, allowing the replication strategy to adapt rather than maintaining static high replication for all scenarios
2Reliability
If traditional data distribution algorithms are used to map data to storage nodes, then data redundancy is increased, but operating costs increase
Solution Approach 1:
The patent modifies the replication parameter by using RDF to control the size of candidate resource sets, enabling the system to achieve adequate redundancy with fewer replicas and thus lower operating costs
3Device complexity
If a single node shares data with many other nodes, then data distribution is simplified, but the probability of data loss increases upon hardware or software failure
Solution Approach 1:
The patent applies local quality by creating distinct candidate resource sets for different data objects based on their access patterns and failure scenarios, rather than using a uniform distribution approach across all data
Solution Approach 2:
The system dynamically determines the number of candidate nodes (RDF) based on current system conditions, allowing the data distribution to adapt to failure scenarios and reduce data loss probability without excessive complexity
4Reliability
If data is rebalanced among remaining nodes after failure, then data availability is restored, but system performance degrades significantly
Solution Approach 1:
The patent performs preliminary actions by pre-calculating and storing candidate resource sets for each data object before failures occur. When failures happen, the system can quickly select from pre-computed candidates rather than performing complex real-time rebalancing, thus maintaining performance while restoring availability
Data Source
AI summary
A system and method for data storage. The method can include: identifying, by a computer processor, a cluster map representing a set of storage resources; for each storage resource of the set of storage resources: traversing, by the computer processor, the cluster map to map the storage resource to a candidate resource set including at least one other storage resource of the set of storage resources; identifying a first data object associated with a storage request; identifying a first candidate resource set based on the first data object; and selecting a first final resource set based at least on the first candidate resource set, where the first data object is sent to storage resources of the first final resource set for storage.