Filesystem Block Grouping for Small-File Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional fixed-block-size filesystems waste storage space when storing files or metadata smaller than the fixed block size, leading to inefficiencies in data storage capacity utilization.
Innovation Solution
Implementing a reference count system in a filesystem free space bitmap to group multiple small data files and metadata structures into a single filesystem block, enabling deduplication and reducing the number of used blocks, thereby improving storage density and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If a fixed-block-size filesystem is used to store files and metadata, then storage allocation is simplified and management is easier, but storage space is wasted when files or metadata are smaller than the fixed block size
Solution Approach 1:
The patent merges multiple small files and metadata structures into a single filesystem block by grouping them together. The filesystem allocator groups multiple small objects (files or metadata) that would individually not fill a block, and stores them together in one block, thereby eliminating the wasted space that would occur if each object were stored in its own separate block.
2Ease of operation
If multiple small files are stored separately in individual filesystem blocks, then file access is simpler and more direct, but the number of used blocks increases and storage density decreases
Solution Approach 1:
The patent combines multiple small files into a single filesystem block, reducing the total number of blocks required. The filesystem maintains references to the grouped files within the block, allowing access to individual files while utilizing shared block space, thereby decreasing storage fragmentation and improving overall storage density.
3Device complexity
If each metadata structure is stored in a separate filesystem block, then metadata management is more straightforward, but storage capacity is wasted and the load on the storage system increases
Solution Approach 1:
The patent groups multiple metadata structures together in a single filesystem block, reducing the number of blocks required for metadata storage. The filesystem maintains the ability to manage and access individual metadata structures while benefiting from the space efficiency of consolidated storage, thereby reducing wasted capacity and decreasing the load on the underlying storage system.
4Reliability
If the filesystem allocates separate blocks for each small object, then object isolation is better and recovery is easier, but storage efficiency decreases and throughput is reduced
Solution Approach 1:
The patent combines multiple small objects into shared filesystem blocks while maintaining logical separation and individual accessibility. The filesystem structure preserves the ability to isolate and recover individual objects through reference tracking, while the physical consolidation improves storage efficiency and reduces the number of I/O operations required, thereby increasing throughput without sacrificing reliability.
Data Source
AI summary
In some examples, a computing device stores root metadata data structures (DSs) together in a group root metadata filesystem block, and stores, in a data filesystem block, at least first file data referenced by the first root metadata DS and second file data referenced by a second root metadata DS. The computing device may determine a reference count of the data filesystem block based on the first file data referenced by the first root metadata DS and the second file data referenced by the second root metadata DS. In addition, the computing device may determine a third file data having content that matches the first file data and referenced by a third root metadata DS. The computing device may update the third root metadata DS to reference the first file data, and update the reference count of the data filesystem block based on updating the third root metadata DS.