Filesystem Deduplication Using Variable Length Sharing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing filesystem deduplication methods for primary storage face performance issues due to high resource requirements and inefficiencies in identifying and eliminating duplicate data, particularly when files are modified by a single byte, making them unsuitable for live storage environments.
Innovation Solution
The implementation of a filesystem deduplication system using variable length sharing with small fingerprint granularity, which allows for periodic deduplication and efficient file geometry management by identifying and eliminating duplicate data portions through a kernel-based process, reducing the need for extensive memory usage and minimizing performance penalties.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If file level deduplication is performed, then storage space can be saved, but significant resources are required for comparing files
Solution Approach 1:
The patent segments files into fixed-size blocks and performs deduplication at the block level rather than file level. Each block is independently hashed and compared, allowing storage savings without requiring resources proportional to entire file comparisons. This segmentation enables efficient handling of modified files by only processing changed blocks.
Solution Approach 2:
The patent replaces mechanical file comparison operations with cryptographic hashing. Instead of comparing file contents byte-by-byte, the system computes hashes of file blocks and compares hash values, dramatically reducing computational resources while maintaining deduplication effectiveness.
2Loss of substance
If data is compared before writing to disk, then duplicate data can be eliminated, but application performance is negatively impacted
Solution Approach 1:
The patent performs deduplication as a preliminary action before data is written to disk. By computing hashes and identifying duplicates during the write operation itself, the system eliminates duplicate data before it occupies storage space, avoiding performance penalties from post-write deduplication while maintaining application performance.
Solution Approach 2:
The patent extracts the deduplication logic from the application write path and implements it as a separate preprocessing step. The application writes data without being blocked by deduplication operations, while the deduplication system independently processes incoming data streams, removing the performance penalty from the application's perspective.
3Productivity
If large amounts of memory are used to prevent lookups, then performance penalty is reduced, but this is not practical in most systems
Solution Approach 1:
The patent changes the parameter of hash table size from large (requiring extensive memory) to a more practical size by using fixed-size block hashing. The system uses cryptographic hashes of fixed-size blocks as keys, allowing efficient memory usage while maintaining fast lookup performance through the properties of cryptographic hash functions.
Data Source
AI summary
Embodiments of the present invention are directed to a method and system for filesystem deduplication that uses both small fingerprint granularity and variable length sharing techniques. The method includes accessing, within an electronic system, a plurality of files in a primary storage filesystem and determining a plurality of fingerprints for the plurality of files. Each respective fingerprint may correspond to a respective portion of a respective file of the plurality of files. The method further includes determining a plurality of portions of the plurality of files where each of the plurality of portions has the same corresponding fingerprint and accessing a list comprising a plurality of portions of files previously deduplicated. A portion of a file of the plurality of files not present in the list may then be deduplicated. Consecutive portions of variables lengths having the same corresponding fingerprints may also be deduplicated.