Size-Based Metadata Storage to Reduce Index Redundancy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems generate excessive index metadata when handling large data reads, consuming storage resources and affecting query or read efficiency.
Innovation Solution
Store metadata based on the size of the target data, avoiding the need to split it into multiple pieces, and utilize a metadata storage index framework (MGTree) with varying data granularities to reduce metadata redundancy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If data is split into fixed-size granularity units for storage, then mapping relationship management is simplified, but excessive index metadata is generated consuming storage resources
Solution Approach 1:
The patent segments the index metadata storage into multiple levels: hot metadata stored in memory for frequent access, warm metadata stored in SSD for moderate access, and cold metadata stored on disk for infrequent access. This hierarchical segmentation reduces the impact of metadata volume on overall system performance while maintaining manageable mapping relationships at each level.
Solution Approach 2:
The patent introduces a new dimension of storage hierarchy (memory-SSD-disk) beyond the traditional single-level storage. By adding the memory dimension for hot metadata and SSD dimension for warm metadata, the system manages large volumes of index metadata without proportionally increasing access complexity, as different dimensions handle different access patterns.
2Reliability
If large amount of index metadata is generated for data read operations, then comprehensive data mapping is achieved, but storage resources are excessively consumed
Solution Approach 1:
The patent extracts hot metadata (frequently accessed index metadata) from the main storage system and places it in memory. This extraction reduces the amount of metadata that needs to be maintained in persistent storage, thereby reducing storage resource consumption while maintaining complete data mapping capabilities through the combination of extracted hot metadata and remaining stored metadata.
Solution Approach 2:
The patent implements a lifecycle management strategy where metadata is moved from hot to warm to cold storage based on access patterns. Metadata that is no longer frequently accessed is effectively 'discarded' from high-resource storage (memory) and moved to lower-resource storage (SSD or disk), reducing overall storage resource consumption while preserving the complete mapping information throughout its lifecycle.
3Device complexity
If metadata is stored in fixed-size units, then storage structure is simplified, but data read/write efficiency is reduced due to excessive index nodes
Solution Approach 1:
The patent implements dynamic metadata management where the storage location and access path for metadata are not fixed but adapt based on access patterns. Hot metadata dynamically resides in memory for fast access, while cold metadata dynamically moves to disk storage. This dynamic approach maintains simple storage structures at any given moment while optimizing read/write efficiency through adaptive metadata placement.
Solution Approach 2:
The patent merges multiple storage technologies (memory, SSD, disk) into a unified metadata storage hierarchy. This combination allows the system to maintain the simplicity of fixed-size storage units while achieving high read/write efficiency by accessing the most relevant metadata from the fastest storage medium, effectively merging the advantages of different storage types.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments of this application disclose a metadata storage method. The method may be applied to a storage system. The method in embodiments of this application includes: providing a first metadata storage unit and a second metadata storage unit, where a size of the first metadata storage unit is greater than a size of the second metadata storage unit; and when a size of target data reaches a specified threshold, storing metadata of the target data based on the first metadata storage unit; or when the size of the target data is less than the specified threshold, storing the metadata of the target data based on the second metadata storage unit. Embodiments of this application can reduce corresponding metadata during data storage and save storage resources.