Composite Data Storage Layout for Small File Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file systems are inefficient in storing small files, as they often occupy entire disk blocks, leading to significant space wastage and inefficiency, and lack flexibility in dynamically managing data layouts, which complicates the storage and retrieval of small data objects.
Innovation Solution
Implementing a composite data storage layout that utilizes a key-value storage system for the first extent of each data object, while storing additional extents in a traditional storage system, with a hierarchical structure of sub-layouts to efficiently manage and dynamically remap data locations, allowing for flexible storage and retrieval of small files.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If traditional storage systems store small files in entire disk blocks, then storage capacity is maintained, but storage space is wasted significantly
Solution Approach 1:
The patent segments data objects into extents of varying sizes (e.g., first extent of 4KB, second extent of 512 bytes, third extent of 128 bytes). This segmentation allows small files to occupy only the necessary storage space rather than entire disk blocks, directly resolving the contradiction between storage space efficiency and layout complexity by enabling precise space utilization through hierarchical extent management.
2Loss of energy
If small files are stored efficiently, then storage space is optimized, but retrieval speed may be reduced
Solution Approach 1:
The patent applies local quality by assigning different storage characteristics to different extents based on their access patterns and size requirements. Critical extents (smaller, frequently accessed) are stored in faster key-value storage systems, while less critical extents are stored in traditional storage systems. This resolves the contradiction by optimizing both storage space utilization and retrieval speed through location-specific storage quality.
3Adaptability or versatility
If a single storage system is used, then system simplicity is maintained, but flexibility in managing different data sizes is reduced
Solution Approach 1:
The patent implements universality by creating a composite storage layout that functions across multiple storage systems (key-value storage and traditional storage). The layout manager provides a unified interface that dynamically selects appropriate storage locations based on extent characteristics, enabling the system to handle various data sizes and access patterns flexibly while maintaining a single logical view of storage, thus resolving the contradiction between adaptability and complexity.
Data Source
AI summary
A composite layout to store one or more extents of a data object in a first storage system and one or more extents of the data object in a second, different storage system. The first storage system may be configured for the efficient storage of small chunks of data such as, e.g., chunks of data small than the addressable block size of the storage devices used by the storage systems.