Lock Metadata Re-balancing During Storage Expansion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing scale-out storage systems face challenges in re-balancing metadata across newly added storage nodes during expansion, requiring minimal interruption to ongoing storage processes and maintaining performance.
Innovation Solution
A method and system for metadata re-balancing that involves an expansion manager coordinating compute entities to update lock and mapping metadata structures across new storage nodes using translation functions, allowing seamless transition to expanded metadata storage space without interrupting storage operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If metadata re-balancing is performed during storage expansion, then storage capacity and performance are improved, but system interruption and latency increase
Solution Approach 1:
The system performs preliminary actions by calculating new metadata addresses and preparing translation functions before the actual expansion operation. Compute entities are instructed in advance about address mapping changes, allowing them to prepare for the transition. This preliminary preparation minimizes the actual interruption time during expansion by having all address translation mappings ready before data migration begins.
Solution Approach 2:
The system maintains continuity of storage operations during metadata re-balancing by implementing dual address space support. Compute entities can continue accessing metadata using old addresses while simultaneously learning new address mappings. The expansion manager coordinates this transition so that storage operations continue uninterrupted, with metadata accessible throughout the expansion process through address translation.
2Productivity
If metadata is re-balanced across new storage nodes, then storage efficiency is improved, but system complexity increases
Solution Approach 1:
The expansion manager acts as an intermediary between compute entities and the expanded storage infrastructure. It coordinates the metadata re-balancing process, manages address translation functions, and synchronizes compute entities with new addressing schemes. This intermediary layer abstracts the complexity of metadata redistribution from individual compute entities, centralizing the complex coordination logic in the expansion manager while keeping node-level operations relatively simple.
Solution Approach 2:
The system changes addressing parameters to accommodate expanded storage. Translation functions map old metadata addresses to new addresses in the expanded address space, allowing the same compute entities to access metadata with modified address parameters. This parameter transformation approach allows metadata to be redistributed across new storage nodes without fundamentally changing the compute entity architecture or access patterns.
3Reliability
If lock metadata structures are updated during expansion, then data consistency is improved, but deadlock risk increases
Solution Approach 1:
The system performs preliminary lock acquisition on both old and new metadata addresses before completing the transition. Compute entities acquire locks in advance on the old address space, then acquire corresponding locks on the new address space before switching. This preliminary locking sequence ensures that no other compute entity can access the metadata during the transition, preventing deadlocks while maintaining data consistency throughout the re-balancing process.
Data Source
AI summary
A method for lock metadata re-balancing upon an expansion of a storage system, the method may include instructing multiple compute entities, by an expansion manager and during a first phase of the lock metadata re-balancing, to acquire locks by using: (a) a current translation function for linking addresses of current lock metadata units (LMUs) to identifiers of stored entities; wherein the current translation function is based on a size of a current storage space allocated to the lock metadata; and (b) a next translation function for linking addresses of new LMUs to identifiers of stored entities; wherein the new translation function is based on a size of a new storage space allocated to the lock metadata; wherein the current space is expanded during the expansion of the storage system to provide the next storage space. Instructing the multiple compute entities, by the expansion manager, and during another phase of the lock metadata re-balancing that follows a release of all current LMUs, to acquire locks and check status of locks using the next translation function without using the current translation function.