Dropped Write Detection via Monotonic Counter Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems fail to detect and correct dropped writes, leading to data corruption and restoration difficulties, even in the absence of storage volume failures.
Innovation Solution
The implementation of a method that adds a monotonically increasing counter value to metadata in write commands, allowing for detection and correction of dropped writes by comparing counter values across mirrored storage systems, and using RAID-level protection to generate and verify metadata instances.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional data storage systems use standard mirroring without extended metadata, then device complexity is reduced, but dropped writes cannot be detected leading to data corruption
Solution Approach 1:
The metadata is segmented into standard portions and extended portions. The extended metadata includes a monotonically increasing counter value that is specifically designed for dropped write detection, separating the detection function from the standard metadata structure.
Solution Approach 2:
A monotonically increasing counter value is introduced as an intermediary element in the extended metadata. This counter serves as a mediator that enables comparison between source and target storage systems to detect dropped writes without requiring complex structural changes to the entire metadata system.
2Reliability
If no counter value is added to metadata, then write operation speed is maintained, but dropped writes remain undetected causing data corruption
Solution Approach 1:
The monotonically increasing counter value is preliminarily incremented and embedded in the extended metadata before the write operation completes. This preliminary action enables subsequent detection of dropped writes by comparing counter values between source and target systems.
Solution Approach 2:
A new parameter (monotonically increasing counter value) is added to the metadata structure. This parameter change enables the system to track write operations and detect dropped writes through simple counter value comparison without fundamentally changing the metadata processing workflow.
3Ease of repair
If extended metadata with counter values is implemented, then dropped writes can be detected and corrected, but metadata processing complexity increases
Solution Approach 1:
The system implements feedback by reading back the counter value from the target storage system and comparing it with the source counter value. This feedback mechanism automatically detects dropped writes and triggers correction operations, simplifying the restoration process despite the added metadata complexity.
Solution Approach 2:
The extended metadata with counter values enables the storage system to self-diagnose and self-correct dropped writes. By automatically comparing counter values and identifying discrepancies, the system performs self-service repair operations without requiring complex external intervention.
Data Source
AI summary
Implementations for dropped write detection and correction are described. An example method includes receiving a write command comprising data and associated metadata; increasing a value of a monotonic counter; generating updated metadata by adding the counter value to the metadata; atomically writing (a) the data and a first instance of the updated metadata to a first storage device, and (b) a second instance of the updated metadata to a second storage device; receiving a read request for the data; reading the first instance of the updated metadata from the first storage device; reading the second instance of the updated metadata from a second storage device; comparing the instances of metadata and the counter values within each instance of metadata; determining whether the first counter value matches the second counter value; and determining whether a dropped write has occurred based on whether the first counter values matches the second counter value.