Storage Controller Metadata Loading During Restart
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
During a storage system restart, metadata loading can exceed time constraints, leading to IO request timeouts and inefficient resource allocation, as the 'Lazy Load' background process loads metadata in a predetermined order, often loading unnecessary data, which can waste processing resources and result in repeated timeouts for IO requests requiring metadata not yet loaded.
Innovation Solution
The storage controller generates an IO thread and a detachable loading thread to prioritize metadata loading during restarts, allowing the loading thread to continue even if the IO thread times out, and marks critical metadata buckets as 'hot' for prioritized loading, ensuring timely fulfillment of IO requests and efficient resource allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If metadata is loaded using background processes in a predetermined order (Lazy Load), then the storage controller can maintain a simple loading mechanism, but the metadata loading time exceeds the hard time constraint for IO requests, causing timeouts
Solution Approach 1:
The patent creates a loading thread in advance when an IO request is received, before the actual metadata loading begins. This loading thread is dedicated to loading the specific metadata needed for the IO request and can operate with higher priority than background processes, ensuring the metadata is loaded within the hard time constraint while maintaining a relatively simple overall mechanism
Solution Approach 2:
The patent dynamically adjusts the loading mechanism by creating a dedicated loading thread based on the specific needs of each IO request. Instead of a static predetermined loading order, the system dynamically prioritizes loading metadata that is immediately needed for servicing IO requests, while less critical metadata continues to be loaded in the background
2Productivity
If metadata is loaded on demand when an IO request is received, then the metadata loading can be prioritized for current IO requests, but the loading time exceeds the predetermined period of time, resulting in IO request timeouts
Solution Approach 1:
The patent segments the metadata loading process into two distinct threads: a loading thread created specifically for loading metadata needed by an IO request, and the IO thread that services the request. The loading thread can be detached from the IO thread, allowing it to continue loading metadata even if the IO request times out, ensuring that the loading operation completes reliably while the IO request can still be serviced or timed out as appropriate
Solution Approach 2:
The loading thread is designed to continue its useful action of loading metadata even after the IO request times out. By detaching the loading thread from the IO thread, the system ensures that the metadata loading process is not aborted due to IO timeout, maintaining continuous and reliable metadata loading operations that improve overall system productivity
3Reliability
If the loading thread is detached from the IO thread upon timeout, then the metadata loading can continue completion, but the IO thread must handle the timeout and retry logic
Solution Approach 1:
The loading thread acts as an intermediary between the IO request and the metadata storage system. It is created to handle the specific task of loading metadata, and when the IO request times out, the loading thread can be detached and continue its operation independently. This intermediary approach allows the system to maintain reliable metadata loading completion while managing timeout and retry logic in a structured way, with the loading thread serving as a mediator that can operate independently of the IO request lifecycle
Data Source
AI summary
During a restart process in which metadata is loaded from at least one of a plurality of storage devices into a cache, a storage controller is configured to generate an IO thread in response to the receipt of an IO request, identify at least one metadata page of the metadata that is used to fulfill the IO request, and generate a loading thread in association with the received IO thread that is configured to cause the storage controller to perform prioritized loading of the identified at least one page of the metadata into the cache. The loading thread is detachable from the IO thread such that, in response to an expiration of the IO thread, the loading thread continues to cause the storage controller to perform the prioritized loading until the loading of the at least one page of the metadata into the cache is complete.