Deduplicating File System Metadata Storage Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication systems inefficiently compare large volumes of data, wasting resources by directly comparing data segments against millions of stored segments, and lack effective strategies for managing metadata storage across distributed nodes.
Innovation Solution
A distributed and deduplicating file system that uses fingerprinting (e.g., SHA-1 hash) to identify unique data segments, optimally stores frequently accessed metadata in flash memory, and writes larger metadata segments to object storage, buffering data in a distributed log to efficiently manage storage and reduce latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a deduplication system directly compares data file segments against previously stored segments, then duplicate detection accuracy is improved, but system resource consumption increases
Solution Approach 1:
The patent introduces fingerprints as an intermediary mechanism between data segments. Instead of directly comparing large data segments, the system generates fingerprints (hash values) that serve as compact representatives. These fingerprints act as mediators that enable efficient comparison while preserving the ability to accurately identify duplicates, thus resolving the contradiction between detection accuracy and resource consumption.
Solution Approach 2:
The patent extracts the essential identifying特征 of data segments by generating fingerprints that represent the entire segment. This extraction process separates the critical identification information (fingerprint) from the full data segment, allowing comparisons to be performed on the extracted fingerprint rather than the complete segment, thereby reducing computational resources while maintaining duplicate detection accuracy.
2Speed
If metadata is stored in flash memory for fast access, then access speed is improved, but storage capacity is limited
Solution Approach 1:
The patent segments metadata into different categories based on access frequency and size requirements. Frequently accessed, small metadata items are stored in flash memory for fast access, while less frequently accessed or larger metadata is stored in object storage. This segmentation allows the system to optimize for speed where needed while managing limited flash memory capacity effectively.
Solution Approach 2:
The patent applies different storage qualities to different metadata based on their specific access patterns and size requirements. Critical, frequently accessed metadata receives the high-quality fast storage of flash memory, while other metadata uses the lower-cost object storage. This local quality differentiation resolves the contradiction by allocating premium storage resources only where they provide the most value.
3Reliability
If all backup data is stored to maintain complete restoration capability, then data protection reliability is improved, but storage utilization deteriorates
Solution Approach 1:
The patent uses fingerprint copying as a reference mechanism. Instead of storing complete duplicate data segments, the system stores a compact fingerprint copy that represents the segment. When restoration is needed, the system can verify data integrity through these fingerprint copies while storing only one actual copy of the original data segment, thereby maintaining reliability while improving storage utilization through deduplication.
Data Source
AI summary
A distributed and deduplicating file system that stores backup metadata to object storage is described. A node, in a cluster of nodes, receives backup data associated with a write operation to back up a data file, and identifies metadata associated with deduplicated data by deduplicating the backup data. If the access frequency associated with the metadata is greater than or equal to an access threshold, and the size of the metadata is less than or equal to a storage threshold, the cluster writes the metadata to each of the memory devices corresponding to the nodes in the cluster. If the access frequency associated with the metadata is less than the access threshold, or the size of the metadata is more than the storage threshold, the node writes information which includes the metadata to an object storage that stores data objects for the nodes in the cluster.