Small File Storage System Using Data Blocking Scheme
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data storage systems using deduplication face inefficiencies when storing small files, as they incur significant overhead and cost due to the overhead of managing object storage paradigms, particularly in enterprise and cloud computing environments where files like transaction logs, emails, and metadata are common.
Innovation Solution
A novel small file storage system that employs a data blocking scheme with variable and fixed block maps, eliminating the need for metadata segments and reducing processing time, improves read performance by storing small files in larger segments and retaining copies locally to avoid network latency during retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If object storage paradigm with deduplication is used, then storage efficiency for large files is improved, but overhead becomes larger than data for small files
Solution Approach 1:
The patent segments files by size, applying different storage strategies: object storage with deduplication for large files, and traditional file storage for small files. This segmentation resolves the contradiction by matching the storage method to the file size characteristics, avoiding overhead dominance in small file scenarios while maintaining deduplication benefits for large files.
Solution Approach 2:
The patent applies different storage qualities to different file sizes: small files receive traditional file system treatment with metadata optimization, while large files receive object storage treatment with deduplication. This local quality approach allows each file type to be stored with the most appropriate method, resolving the overhead-efficiency contradiction.
2Device complexity
If small files are stored using traditional file storage, then overhead is reduced, but deduplication benefits are lost
Solution Approach 1:
The patent segments the storage system into two paths: traditional file storage for small files to minimize overhead, and object storage with deduplication for large files to maximize storage efficiency. This segmentation allows each file size category to benefit from its optimal storage method.
Solution Approach 2:
The patent changes the storage parameter (method) based on file size parameter. By thresholding on file size, the system transitions between storage modes, applying traditional file storage for files below the threshold and object storage for files above it, thus optimizing both overhead and efficiency for different file sizes.
3Measurement precision
If metadata segments are retained for small files, then retrieval accuracy is improved, but storage capacity is reduced
Solution Approach 1:
The patent extracts metadata management from small file storage by using traditional file system metadata structures instead of object storage metadata segments. This extraction eliminates the redundant metadata overhead while preserving retrieval accuracy through conventional file system metadata mechanisms.
Solution Approach 2:
The patent uses lightweight, conventional file system metadata for small files instead of heavy object storage metadata structures. This disposable metadata approach provides sufficient retrieval accuracy for small files without consuming excessive storage capacity, as the metadata is simple and compact.
Data Source
AI summary
A small file storage system reduces the cost of storing and reading segmented data in a deduplication storage system. A data-blocking scheme is used to store small files instead of generating different levels of metadata as is done for larger files in deduplication systems. Among other advantages, the data blocking scheme reduces the cost of reading metadata segments from the content store, reduces the cost of index lookup, CM region read and decompression, and the cost of recalling and restoring small files from cloud storage, all of which contribute to better read performance. Other advantages include faster verification of small files, and improvements in small file data locality on disk with larger segment sizes. By eliminating the need to update metadata segments, embodiments of the small file storage system advantageously increase the efficiency of file updates, especially since updates to smaller files are more random in nature.