Dynamic Replica Adjustment via Access Probability in Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing distributed file systems, such as HDFS, face inefficiencies in managing the number of replicas of a file, leading to increased storage costs and resource utilization due to a fixed default setting of three replicas, which does not adapt to the varying access probabilities of files.
Innovation Solution
A method and system that dynamically adjust the number of replicas based on access probability by using timers and access patterns, reducing replicas when access is low and increasing them upon write access or high access probability, allowing for optimized storage and resource allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the number of replicas is increased to ensure fast and reliable access, then reliability and access speed are improved, but storage cost and resource utilization increase
Solution Approach 1:
The patent implements dynamic replica management by continuously monitoring file access patterns and adjusting the number of replicas in real-time. Files with high access probabilities maintain more replicas for fast reliable access, while files with low access probabilities have fewer replicas, reducing storage costs. This dynamic adaptation resolves the contradiction between maintaining reliability and reducing storage expenditure.
Solution Approach 2:
The system changes the replica count parameter based on observed access patterns and probability assessments. By calculating access probabilities from historical data and adjusting replica numbers accordingly, the system optimizes the balance between access reliability and storage cost, ensuring that resources are allocated efficiently based on actual file usage characteristics.
2Device complexity
If the number of replicas is set to a fixed default value, then system simplicity is maintained, but storage efficiency and resource utilization deteriorate
Solution Approach 1:
The system automatically monitors its own file access patterns and self-adjusts replica numbers without requiring manual intervention or complex external management. The automated probability calculation and dynamic replica adjustment mechanisms enable the system to optimize storage efficiency while maintaining manageable complexity through rule-based automation.
Solution Approach 2:
The system implements feedback loops by continuously monitoring file access patterns, calculating access probabilities, and using this information to adjust replica numbers. This closed-loop control mechanism enables the system to adapt to changing access patterns and maintain optimal storage efficiency while keeping complexity manageable through automated decision-making based on observed behavior.
3Reliability
If replicas are maintained for all files, then access reliability is ensured, but resource utilization and storage cost increase unnecessarily
Solution Approach 1:
The patent applies different replica management strategies to different files based on their individual access characteristics. High-priority files with frequent access patterns receive more replicas and higher reliability guarantees, while low-priority files with rare access patterns have fewer replicas. This localized, file-specific approach ensures that resource utilization is optimized according to actual access needs rather than applying a uniform strategy to all files.
Data Source
AI summary
In a data storage system with a number of replicas of a file set to one or more replicas, a timer is set to track a time since a last access to the file. Responsive to the timer matching a first timer window threshold, a number of replicas of the file are automatically reduced within the data storage system, wherein the probability that the file will be accessed prior to the first timer window threshold is greater than the probability that the file will be accessed after the first timer window threshold.