Distributed Storage Offline Merging Small Files HDFS
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing Hadoop Distributed File System (HDFS) faces inefficiencies when storing small files, leading to high memory overheads for the NameNode and low computing performance for MapReduce tasks due to its design primarily for large files, resulting in excessive memory usage and inefficient data access.
Innovation Solution
A distributed storage method that involves offline merging of small files into larger files using a first thread service, generating metadata to indicate their locations, and utilizing a second thread service to manage and perform operations on these files, reducing memory overheads and improving computing efficiency by optimizing metadata management and MapReduce performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If small files are stored individually in HDFS, then file access simplicity is maintained, but memory overhead of NameNode increases significantly
Solution Approach 1:
Multiple small files are merged into a single archive file (e.g., TAR format) stored in HDFS. The NameNode stores metadata for the archive file instead of individual small files, reducing memory overhead while maintaining the ability to access individual small files through the archive structure.
Solution Approach 2:
Small files are nested within an archive file structure. Each small file is packaged inside the archive, allowing the NameNode to manage the archive as a single entity while preserving the hierarchical structure and metadata of individual small files within the archive.
2Device complexity
If small files are stored individually in HDFS, then individual file metadata is easily managed, but MapReduce computing performance deteriorates
Solution Approach 1:
Multiple small files are combined into a single archive file, reducing the number of metadata entries the NameNode must manage. MapReduce tasks can process the archive file as a single unit or extract and process individual files in parallel, improving computing performance by reducing task overhead and improving data locality.
3Quantity of substance
If small files are merged into archive files, then memory overhead is reduced, but file operation complexity increases
Solution Approach 1:
A file system translation layer or gateway is introduced between the user/application and the archived small files in HDFS. This intermediary handles the complexity of archive file operations (creation, extraction, modification) while presenting simple file operations to users, thus reducing memory overhead without significantly increasing user-facing operation complexity.
4Stability of the object's composition
If traditional HDFS stores each small file separately, then data locality is maintained, but redundant data increases
Solution Approach 1:
Multiple small files with redundant or related content are merged into a single archive file stored in HDFS. This eliminates redundant data storage while maintaining data locality, as the archive file can be stored on the same DataNode where the original small files resided, allowing efficient local access without duplicating data across multiple file entries.
Data Source
AI summary
A distributed storage method includes offline merging, by a first thread service of a distributed storage system, M small files in a file system; generating, by the first thread service, M pieces of metadata in the offline merging process; loading, by a second thread service of the distributed storage system, the M pieces of metadata into a metadata set; searching, by the second thread service, the metadata set for metadata of a first small file when the second thread service receives a first instruction; and performing, by the second thread service, the operation corresponding to the first instruction when the second thread service finds the metadata of the first small file in the metadata set.