Dual Layer Deduplication File System Block Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Block-based deduplication storage systems face inefficiencies in achieving effective deduplication ratios due to block alignment issues, particularly when dealing with files that have minor differences, such as videos with embedded subtitles, leading to reduced deduplication effectiveness compared to byte-aligned systems.
Innovation Solution
Implementing a dual-layer deduplication system where a file system driver breaks files into chunks and adds trailing zeros to align with block sizes, enabling efficient deduplication by leveraging both file and block-based storage systems, thereby enhancing deduplication ratios with minimal performance impact.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If block-based deduplication is used, then storage efficiency is improved through eliminating duplicate blocks, but deduplication effectiveness deteriorates due to block alignment issues when files have minor differences
Solution Approach 1:
The patent segments files into variable-length chunks rather than fixed-size blocks, allowing the system to capture complete semantic units (such as entire video files with subtitles) as single deduplication units. This segmentation approach enables the file system driver to break files at optimal points that preserve data integrity while maximizing deduplication opportunities across similar files.
Solution Approach 2:
The patent introduces a dual-layer deduplication architecture that operates at both the file system level (chunking files into variable-length segments) and the block storage level (fixed-size block deduplication). This multi-dimensional approach allows the system to leverage both variable-length semantic chunking and fixed-size block alignment, resolving the contradiction between storage efficiency and deduplication effectiveness by operating in multiple deduplication dimensions simultaneously.
2Manufacturing precision
If variable-length chunking is used, then deduplication ratios are improved by capturing complete semantic units, but storage space increases due to lack of block alignment
Solution Approach 1:
The patent applies preliminary padding with trailing zeros to variable-length chunks before storing them in the block storage system. This preliminary action ensures that all data units are aligned to the block storage system's native block size boundaries, eliminating wasted space from misalignment while preserving the variable-length chunking benefits for deduplication. The padding is minimal and systematic, ensuring optimal space utilization.
Solution Approach 2:
The patent dynamically adjusts chunk sizes based on the content being stored, allowing variable-length chunks to adapt to different file types and structures. By changing the chunk size parameter to match semantic boundaries (such as complete video files or document sections), the system achieves higher deduplication ratios for similar files while the trailing zero padding ensures these variable chunks align with fixed block boundaries, preventing storage space waste.
Data Source
AI summary
An aspect of dual layer deduplication for a file system running over a deduplication block storage system includes accessing a file by a file system driver of the file system. The file is stored as one of a plurality of files in the file system. A further aspect includes breaking the file down into multiple blocks and generating chunks from the blocks. For at least one of the chunks, a trail of zeros is added until a size of a respective one of the chunks is a multiple of a block size of the deduplication block storage system.