Packed Block Storage for Sub-4K Data Chunks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current mass data storage systems inefficiently utilize physical storage space due to allocating entire 4K blocks for files, even if they contain only a few bytes of valid data, leading to wastage, especially in small files and compressible data.
Innovation Solution
A file system protocol that identifies sub-4K data chunks, collects and packs them into a single 4K block, using metadata to store offset, length, and lost-write context, allowing multiple blocks to share the same disk space and efficiently utilize storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional block-based file system allocates entire 4K blocks for files, then data storage is simplified and reliable, but physical storage space is wasted when files contain only a few bytes of valid data
Solution Approach 1:
The patent segments the fixed 4K block structure into variable-length sub-blocks that can accommodate actual data sizes. Instead of treating each 4K block as a single atomic unit, the system divides blocks into smaller segments based on actual data requirements, allowing efficient packing of multiple small files into one physical block while maintaining the reliability of block-based storage.
Solution Approach 2:
The patent merges multiple sub-4K data chunks from different files into a single 4K physical block. By combining multiple small data segments that would traditionally require separate 4K blocks, the system achieves space efficiency while maintaining data integrity through proper metadata tracking and offset management.
2Speed
If multiple small files are stored using traditional 4K block allocation, then file access is simple and fast, but disk space consumption is excessive
Solution Approach 1:
The patent performs preliminary actions by pre-calculating and storing metadata information (offsets, lengths, chunk identifiers) during the data writing phase. This preliminary organization of data and metadata enables fast random access to individual chunks within packed blocks without requiring full block reads, thus maintaining access speed while reducing space consumption.
Solution Approach 2:
The patent changes the parameter of block allocation from fixed 4K per file to variable-length sub-blocks within a 4K physical block. By dynamically adjusting the size and positioning of data chunks based on actual file requirements, the system achieves both fast access (through proper indexing) and efficient space utilization.
3Ease of operation
If the file system uses fixed 4K block addressing, then storage management is straightforward, but storage efficiency deteriorates for compressible data and small files
Solution Approach 1:
The patent introduces metadata structures as intermediaries between the file system and physical storage blocks. These metadata structures track the organization of data chunks within packed blocks, providing a layer of abstraction that maintains straightforward storage management while enabling efficient space utilization through variable-length chunk packing.
Solution Approach 2:
The patent adds a new dimension to block addressing by introducing offset and length parameters within the 4K block structure. Instead of simple block-number-based addressing, the system uses a multi-dimensional addressing scheme that includes block identifier, offset position, and data length, enabling efficient packing while maintaining manageable storage operations.
Data Source
AI summary
Presented herein are mass data storage networks, file system protocols, non-transitory machine readable devices, and methods for storing data blocks in mass data storage systems. Methods for storing data blocks in a file system are disclosed which include: receiving by storage controller of the data storage system a request to write a data file to a system storage module; determining whether the data file includes a sub-K data chunk that is less than approximately four kilobytes; identifying a packed block that stores a plurality of sub-K data chunks and has sufficient storage space available to store the sub-K data chunk; and placing, by the storage controller in the packed block, the sub-K data chunk and a corresponding data length and a respective offset identifying a location of the sub-K data chunk in the packed block.