Fair Data Scrubbing in Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face challenges in ensuring data durability and integrity due to hard disk errors, such as latent sector failures and silent corruption, where replication alone may not be sufficient to prevent data loss, and disk scrubbing processes are slow and inefficient, particularly when prioritizing the order of data verification.
Innovation Solution
Implementing a fair data scrubbing technique that balances the verification of recently stored data with minimizing latency by tracking and prioritizing 'never-scrubbed' and 'already-scrubbed' data, adjusting the scrubbing ratio based on thresholds and monitoring data storage media device activity to ensure timely detection and correction of errors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If disk scrubbing is performed continuously at high speed, then data verification coverage is improved, but impact on other disk access processes increases
Solution Approach 1:
The system implements periodic disk scrubbing by allocating a capped disk bandwidth to the scrubbing process rather than running continuously at full speed. This periodic action allows the scrubbing to occur at controlled intervals that balance verification needs with normal disk access requirements.
Solution Approach 2:
The system dynamically adjusts the scrubbing rate based on system conditions. By monitoring disk bandwidth availability and adjusting the scrubbing speed accordingly, the system can accelerate scrubbing when resources are available and slow down when other processes need disk access, resolving the contradiction between verification coverage and impact on other processes.
2Ease of operation
If disk scrubbing is performed slowly with capped bandwidth, then impact on other processes is reduced, but data verification time increases
Solution Approach 1:
The scrubbing process dynamically adjusts its bandwidth allocation based on system conditions and data priority. When disk bandwidth is available and data has been quiesced for longer periods, the scrubbing rate increases to reduce overall verification time while still protecting other processes during peak usage periods.
Solution Approach 2:
The system prioritizes scrubbing data that has been quiesced for longer periods before scrubbing more recently accessed data. This preliminary action on older data reduces the window of vulnerability for the most at-risk data first, effectively reducing overall verification time for critical data while maintaining capped bandwidth to protect other processes.
3Reliability
If recently written data is scrubbed immediately, then data correctness is improved, but latency between writing and scrubbing increases
Solution Approach 1:
The system performs preliminary scrubbing on data that has been quiesced for longer periods before scrubbing recently written data. This approach ensures that the most vulnerable data (older quiesced data) is verified first, improving data correctness for critical data while minimizing latency for recently written data that is still in faster storage tiers.
Solution Approach 2:
The system applies different scrubbing priorities to different data based on their characteristics. Recently written data in fast storage tiers receives lower scrubbing priority with shorter latency, while older data in slower tiers receives higher priority scrubbing, optimizing the balance between data correctness and latency for each data segment.
4Reliability
If the entire multi-terabyte hard disk is scrubbed, then data safety is improved, but scrubbing duration increases to weeks
Solution Approach 1:
The system segments the scrubbing process by dividing the disk into different data categories based on quiescence time and access patterns. By scrubbing segments of data (older quiesced data first, then progressively newer data) rather than the entire disk uniformly, the system maintains data safety through comprehensive verification while reducing the perceived duration by completing critical verifications sooner.
Solution Approach 2:
The system implements periodic scrubbing cycles that systematically work through different portions of the disk over time. This periodic approach ensures complete data safety coverage across the entire multi-terabyte disk while managing the total scrubbing duration by distributing the verification load across multiple cycles rather than attempting to scrub everything continuously at once.
Data Source
AI summary
Computer-implemented techniques for fair data scrubbing. The techniques can be used to balance a desire to verify recently stored data soon after it is stored on a target data storage media device, when the computing and networking cost of reconstructing the data in the event of a detected data storage media device error can be lower, against a desire to minimize the latency between rescrubbing data. By doing so, the techniques improve the operation of a data storage system that implements the techniques.