Metadata Page Allocation with Partial In-Memory Bitmaps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional metadata page allocation techniques face challenges in efficiently managing and expanding metadata storage volumes, leading to inefficiencies in processing I/O operations in distributed storage systems.
Innovation Solution
Storing a portion of a metadata volume bitmap in memory to indicate allocation or unallocation of metadata pages, allowing for efficient allocation and deallocation processes without the need for disruptive bitmap expansions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If the entire metadata volume bitmap is stored on disk, then storage space is saved, but allocation and deallocation operations become time-consuming due to frequent disk I/O
Solution Approach 1:
The bitmap is divided into multiple chunks, with only a portion of each chunk stored in memory while the rest remains on disk. This segmentation allows the system to maintain a manageable in-memory bitmap portion for fast allocation operations while storing the complete bitmap on disk for space efficiency.
Solution Approach 2:
The system transitions from storing the entire bitmap in a single location (either fully on disk or fully in memory) to a distributed approach across multiple dimensions (both disk and memory), optimizing for both speed and space by leveraging the different characteristics of each storage medium.
2Adaptability or versatility
If metadata storage volume is expanded, then storage capacity is increased, but the bitmap expansion process becomes disruptive and time-consuming
Solution Approach 1:
The system pre-allocates additional bitmap chunks on disk before they are needed, allowing the bitmap to be expanded by simply adding new chunks to the end of the existing structure. This preliminary preparation eliminates the need for disruptive bitmap reconstruction during volume expansion.
Solution Approach 2:
The bitmap structure is made dynamic and extensible, allowing it to grow automatically as metadata storage volume expands. New bitmap chunks can be appended to the end of the existing bitmap, enabling seamless volume expansion without disrupting ongoing operations.
3Productivity
If the bitmap is stored entirely in memory, then allocation operations are fast, but memory resources are consumed and bitmap expansion becomes complex
Solution Approach 1:
The bitmap is segmented into multiple chunks that are distributed between memory and disk. Only a portion of each chunk is loaded into memory for fast access during allocation operations, while the complete chunk structure remains on disk for manageable complexity and easy expansion.
Solution Approach 2:
Instead of loading the entire bitmap into memory (excessive action) or using a simplified partial bitmap (insufficient action), the system uses a partial loading approach where only the necessary portion of each chunk is in memory, balancing speed requirements with complexity management.
Data Source
AI summary
Techniques are provided for metadata page allocation by storing a portion of a bitmap of metadata pages in memory. One method comprises obtaining a request to allocate metadata pages of a metadata storage volume; identifying at least one bit, corresponding to an unallocated metadata page, in a given chunk of a bitmap of metadata pages associated with the metadata storage volume, wherein the given chunk of the bitmap is stored in memory and comprises a portion of the bitmap; updating a bit value of the identified at least one bit to indicate the allocation of the metadata pages of the metadata storage volume; and initiating an allocation of the metadata pages. The bitmap may be allocated to a designated size, and each bit of the bitmap may indicate whether a corresponding metadata page of the metadata storage volume is allocated or unallocated.