Parallel Log-Structured File System Sparse Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file systems are inefficient in storing sparse files, as they typically process them at the granularity of a file system block, leading to wasteful storage of empty blocks and inefficient data processing and transfer bandwidth.
Innovation Solution
A parallel log-structured file system is used to store sparse files by storing only the data portions and generating index entries with logical and physical offsets, allowing for efficient storage and restoration of holes, with additional efficiency gained by detecting write patterns and storing patterned index entries in a single directory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If existing file systems store sparse files at the granularity of a file system block, then storage allocation is simplified, but disk space is wasted and data processing bandwidth is inefficient
Solution Approach 1:
The patent segments the sparse file storage into two distinct components: a data file containing only actual data blocks and an index file containing metadata about data block locations and file holes. This segmentation allows the system to store only actual data without wasting space on empty blocks, while maintaining simplified storage allocation through the index structure that manages the segmented data blocks.
2Device complexity
If existing file systems allocate blocks for sparse files, then data structure is simplified, but storage efficiency deteriorates
Solution Approach 1:
The patent extracts the metadata information from the traditional block-based file structure and places it in a separate index file. This extraction allows the data file to contain only actual data blocks without empty blocks, improving storage efficiency. The index file independently manages the metadata, maintaining data structure simplicity while enabling efficient storage through the separation of data and metadata.
3Ease of operation
If existing file systems convert metadata to real blocks at runtime, then data access is simplified, but data processing bandwidth is reduced
Solution Approach 1:
The patent performs preliminary actions by pre-calculating and storing the locations of data blocks and file holes in the index file during file creation or data writing. This preliminary organization of metadata eliminates the need for complex runtime conversions, maintaining ease of data access while improving data processing bandwidth by reducing runtime overhead and enabling more efficient data retrieval operations.
Data Source
AI summary
A sparse file is stored without holes by storing a data portion of the sparse file using a parallel log-structured file system; and generating an index entry for the data portion, the index entry comprising a logical offset, physical offset and length of the data portion. The holes can be restored to the sparse file upon a reading of the sparse file. The data portion can be stored at a logical end of the sparse file. Additional storage efficiency can optionally be achieved by (i) detecting a write pattern for a plurality of the data portions and generating a single patterned index entry for the plurality of the patterned data portions; and/or (ii) storing the patterned index entries for a plurality of the sparse files in a single directory, wherein each entry in the single directory comprises an identifier of a corresponding sparse file.