Managed File Compaction for Faster Distributed Small-File Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed file systems face inefficiencies when processing and analyzing small files, leading to slowed processing, increased retrieval times, and wasted storage, as they are not optimized for handling files smaller than a target size or block size.
Innovation Solution
A file-compaction system that monitors storage containers in distributed storage systems, identifies small files, and automatically compacts them into larger files, using a scheduler, inspection, and compaction components to manage and optimize file sizes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If distributed file systems process small files individually, then file processing can be performed, but processing speed decreases and retrieval time increases
Solution Approach 1:
The patent merges multiple small files into a single larger file by creating a master file that contains references to all small files in a directory. This consolidation reduces the number of individual file operations required, thereby improving processing speed and reducing retrieval time while maintaining access to all original small files through the master file's reference structure.
2Quantity of substance
If small files are stored separately in distributed storage systems, then data can be accessed individually, but storage space is wasted due to metadata overhead and fragmentation
Solution Approach 1:
The patent combines multiple small files into a single master file that stores references to all small files. This merging approach reduces storage usage by eliminating redundant metadata and file system overhead that would exist for each small file individually, while maintaining ease of operation through the reference structure that allows access to any small file via the master file.
Solution Approach 2:
The master file acts as an intermediary between the user and the small files. Instead of directly managing multiple small files with their individual metadata, the master file provides a unified interface that references all small files, thereby reducing storage overhead while maintaining access capability.
3Quantity of substance
If a compaction process is initiated to compact small files, then storage efficiency improves, but the storage container must be locked preventing queries during compaction
Solution Approach 1:
The patent performs preliminary actions by creating a master file that references all small files before actual compaction occurs. This allows the system to prepare the reference structure in advance, enabling queries to continue accessing data through the master file during the compaction process, thereby maintaining reliability while improving storage efficiency.
Solution Approach 2:
The master file serves as an intermediary that remains accessible during compaction operations. While small files are being compacted and removed, the master file maintains references to the data, allowing queries to continue without locking the entire storage container, thus preserving data accessibility during the compaction process.
Data Source
AI summary
Techniques implemented by file-compaction system that monitors storage containers in distributed storage systems, identifies small files that have been added to the storage containers, and automatically compacts the small files into larger files. Distributed file systems are well-equipped to manage large data files, but handling small files can slow the processing and analyzing of files, increase the retrieval time of files, etc. The file-compaction system monitors the storage containers to determine whether new small files have been added from data sources. Upon detecting small files that have been added to storage containers, the file compactor automatically compacts those small files into large files. By continuously identifying and compacting small files that have been added to storage containers, the file compactor provides the distributed file system with large files while minimizing the amount of time the storage containers are locked during compaction.