Scrubbing Disaggregated Storage via Local Validation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In big data storage environments like Ceph and Hadoop, the overhead of network traffic during data scrubbing increases significantly when using disaggregated storage targets, as each remote disk's data needs to be read over the network for validation, leading to inefficiencies and potential false positives or negatives due to stale data.
Innovation Solution
Moving the scrubbing process closer to the disk targets, allowing only error data to be sent over the network for verification and correction, thereby reducing the amount of data that needs to be read and minimizing network traffic, and implementing robust validation techniques to handle stale data scenarios.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data scrubbing is performed by reading all remote disk data over the network for validation, then data integrity can be verified, but network overhead increases significantly
Solution Approach 1:
The patent extracts the scrubbing process from the remote host and relocates it to the local host where the disk target is physically located. This allows validation to occur locally without transmitting all data over the network, thereby verifying data integrity while eliminating excessive network overhead
Solution Approach 2:
The patent introduces a scrubbing task as an intermediary component that resides on the local host but operates on remote disk targets. This intermediary performs validation locally and only communicates essential results with the remote host, mediating between the need for verification and the cost of network transmission
2Measurement precision
If all remote disk data is read over the network for validation, then complete data verification is achieved, but network bandwidth is consumed excessively
Solution Approach 1:
The scrubbing operation is extracted from the remote host and executed locally on the host where the disk target resides. This extraction enables complete data validation to occur without transmitting data across the network, achieving measurement precision while eliminating excessive network traffic volume
Solution Approach 2:
The local host performs self-service by executing the scrubbing task locally on its attached disk targets. This self-validation mechanism eliminates the need to read data over the network while maintaining complete verification accuracy, as the host validates its own stored data
3Device complexity
If data scrubbing is performed remotely, then centralized control is maintained, but scrubbing efficiency decreases due to network latency
Solution Approach 1:
The patent segments the scrubbing function from the remote host and assigns it to the local host. This segmentation distributes the scrubbing workload to where the data physically resides, improving productivity by eliminating network latency while maintaining centralized control through the object manager's coordination
4Reliability
If remote scrubbing is implemented, then data validation can occur, but false positives and negatives increase due to stale data
Solution Approach 1:
The local host performs self-validation of its stored data through locally-executed scrubbing tasks. This eliminates the stale data problem entirely, as the host validates data immediately when reading it locally rather than relying on potentially outdated remote copies, thereby improving measurement precision while maintaining validation capability
Data Source
AI summary
Apparatuses, systems and methods are disclosed herein that generally relate to distributed network storage and filesystems, such as Ceph, Hadoop®, or other big data storage environments utilizing resources and/or storage that may be remotely located across a communication link such as a network. More particularly, disclosed are techniques for one or more machines or devices to scrub data on remote resources and/or storage without requiring all or substantially all of the remote data to be read across the communication link in order to scrub it. Some disclosed embodiments discuss having validation be relatively local to storage(s) being scrubbed, and some embodiments discuss only providing to the one or more machines scrubbing data selected results of the relatively local scrubbing over the communication link.