Hierarchical File System Data Block Deduplication via Fingerprint Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face inefficiencies due to duplicate data blocks, which waste storage space, and previous solutions like file folding or hash-based methods either rely on persistent images or impact performance.
Innovation Solution
A method for identifying and eliminating duplicate data blocks in a hierarchical file system by computing fingerprints for each block and storing them in metadata, allowing for sharing without a persistent point-in-time image, and maintaining a reference count file to manage shared blocks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If file folding is used to share data blocks between Snapshot and active file system, then storage space efficiency is improved, but the system requires a persistent point-in-time image (Snapshot) which adds complexity and limits flexibility
Solution Approach 1:
The patent extracts the Snapshot dependency from the data sharing mechanism. Instead of requiring a persistent point-in-time image, the system directly identifies and shares duplicate data blocks between active files using fingerprint comparison, eliminating the need for Snapshot infrastructure while maintaining storage efficiency
Solution Approach 2:
The data sharing mechanism becomes universal by working across all files in the active file system without requiring Snapshot participation. Any duplicate blocks anywhere in the system can be identified and shared through the fingerprint-based approach, making the solution applicable to general file sharing scenarios beyond Snapshot restoration
2Quantity of substance
If hash values are computed for every file to reduce data duplication, then storage space efficiency is improved, but performance deteriorates due to computation overhead on every read/write operation
Solution Approach 1:
The system performs preliminary fingerprint computation only when data blocks are written to storage, not during every read operation. The computed fingerprints are stored with the data blocks, enabling efficient duplicate detection during writes while avoiding performance degradation during reads, as the comparison work was already done during the write phase
Solution Approach 2:
Instead of computing hash values for every file access, the system creates a fingerprint copy of each data block during write operations and stores it in the metadata. This pre-computed fingerprint copy enables rapid duplicate detection without requiring recalculation during subsequent read or write operations, significantly reducing computational overhead
3Loss of substance
If duplicate data blocks are eliminated through traditional methods, then storage space is optimized, but the process requires persistent images which complicates the system architecture
Solution Approach 1:
The patent introduces a metadata structure as an intermediary between data blocks and the sharing mechanism. This metadata stores fingerprints and reference counts, mediating the duplicate detection process without requiring persistent images. The metadata acts as a lightweight index that enables efficient identification and elimination of duplicate blocks while simplifying the overall architecture by removing Snapshot dependencies
Data Source
AI summary
A method for sharing data blocks in a hierarchical file system in a storage server includes allocating a plurality of data blocks in the file system, and sharing data blocks in the file system, without using a persistent point-in-time image, to avoid duplication of data blocks. A method for identifying data blocks that can be shared includes computing a fingerprint for each of multiple data blocks to be written to a storage facility and storing the fingerprint with information identifying the data block in an entry in a set of metadata. The set of metadata is used to identify data blocks which are duplicates.