Object Integrity Service for Silent Data Corruption Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face challenges in detecting and addressing silent data corruption in infrequently accessed objects, which can lead to extensive corruption and service level agreement failures due to resource-intensive data scrubbing processes that disrupt client access and consume system resources.
Innovation Solution
An object integrity service that operates as a background process, scanning and repairing objects stored in a distributed hash table storage system by maintaining a list of unchecked objects, using a message queue to manage tasks, and ensuring minimal resource consumption by capping system resources and bandwidth usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data scrubbing is performed frequently to detect silent data corruption, then data integrity is improved, but system resources and client access are disrupted
Solution Approach 1:
The patent implements periodic data scrubbing by scheduling integrity checks at predetermined time intervals rather than continuously or on-demand. The system maintains a list of data objects with their last checked timestamps, and only performs scrubbing on objects that have exceeded the threshold time period since their last check. This periodic approach ensures data integrity monitoring while minimizing disruption to client access and system resources.
2Reliability
If comprehensive data scrubbing is performed on all objects, then detection of silent data corruption is improved, but resource consumption increases
Solution Approach 1:
The patent applies local quality by differentiating the scrubbing frequency and priority based on individual object characteristics. Rather than uniformly scrubbing all data objects, the system identifies objects that have not been checked within a specific time period and targets only those for scrubbing. This selective approach concentrates resources on objects most likely to have developed silent corruption, improving detection effectiveness while reducing overall resource consumption.
3Quantity of substance
If data objects are not accessed frequently, then storage efficiency is improved, but susceptibility to silent data corruption increases
Solution Approach 1:
The patent implements preliminary action by proactively scheduling integrity checks for data objects based on their last access time and a predefined threshold period. The system maintains metadata tracking when each object was last checked and automatically queues objects for scrubbing when they exceed the time threshold, even if they have not been recently accessed. This preliminary scheduling ensures that infrequently accessed objects receive timely integrity verification before potential corruption spreads or impacts service level agreements.
Data Source
AI summary
An embodiment for object integrity service in a storage system includes generating a list of objects stored in a storage system, wherein the list of objects may list an unchecked object, and wherein the unchecked object is an object that has not been checked within a set time period, walking through the list of objects to identify the unchecked object, adding a task to a queue to check the unchecked object, and clearing the task from the queue by checking the unchecked object.