Storage System Data Consistency Verification via Multi-Pass Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In deduplicated file systems with metadata separation, ensuring data consistency between metadata containers and actual data segments is challenging due to separate storage of fingerprints and data segments, which can lead to uncertainty in identifying all data segments by fingerprints, especially when resources are limited during system maintenance or recovery.
Innovation Solution
A method is implemented where a subset of objects is loaded into memory based on available memory space, and a multi-pass process is performed to verify data consistency, dividing the container set into ranges to optimize memory usage and reduce latency, using metadata containers and object storage with fingerprints to identify data segments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all objects are loaded into memory for data consistency verification, then verification completeness is improved, but memory resource consumption increases
Solution Approach 1:
The patent divides the verification process into multiple passes, where each pass verifies a subset of objects rather than loading all objects simultaneously. This segmentation allows the system to maintain verification completeness while limiting memory usage at any given time to manageable levels.
Solution Approach 2:
The patent implements a multi-pass verification approach where only a partial set of objects is loaded into memory during each pass. By iteratively processing different subsets across multiple passes, the system achieves complete verification without requiring all objects to be in memory simultaneously, thus reducing peak memory consumption.
2Quantity of substance
If a multi-pass process is used to verify data consistency, then memory usage is reduced, but verification time increases
Solution Approach 1:
The verification process is segmented into multiple passes, each handling a specific subset of objects. This segmentation reduces the memory footprint of each individual pass while distributing the total verification time across multiple shorter operations, making the process more manageable and resource-efficient.
Solution Approach 2:
The patent employs periodic action by executing verification in repeated cycles or passes. Each pass completes verification for its assigned subset of objects, releases memory resources, and then the next pass begins. This periodic approach balances memory usage over time while systematically completing the full verification task.
3Productivity
If objects are loaded into memory based on available space, then resource efficiency is improved, but system complexity increases
Solution Approach 1:
The patent implements dynamic object selection based on available memory space. The system adapts its behavior by adjusting which objects are loaded into memory during each pass according to current resource availability. This dynamic approach optimizes resource efficiency while maintaining a relatively simple control mechanism based on memory capacity thresholds.
Data Source
AI summary
According to some embodiment, a storage system provides metadata containers and an object storage that includes one or more objects. Each of the objects includes one or more data segments of a file object and each of the metadata containers includes one or more fingerprints for identifying the data segments in the object. The system loads some of the objects into a memory device of the storage system. For each of the loaded objects, the system traverses some of the metadata containers and performs a lookup operation to determine whether each data segment in the object is identified by a fingerprint from the traversed metadata containers.