On-Demand Metadata Recovery During File System Migration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
During a non-clean shutdown, the metadata file in data storage systems may not accurately reflect the migration state of files, leading to unreliable migration state information and prolonged unavailability due to the time-consuming process of scanning a large metadata file, which can cause excessive delays in user I/O operations.
Innovation Solution
Implementing an on-demand recovery method that allows user I/O operations to proceed before the metadata file scan completes, using a background recovery process, an in-memory map for on-demand recovered entries, and a lock map to manage race conditions, ensuring accurate and non-disruptive metadata recovery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the metadata file is scanned completely to ensure accurate migration state information, then data correctness is improved, but user I/O availability deteriorates due to prolonged data unavailability
Solution Approach 1:
The system performs preliminary actions by starting a background recovery process immediately upon detecting a non-clean shutdown, before user I/O operations are affected. This background process begins scanning the metadata file in the background while user I/O can proceed with on-demand recovery, thus preparing the system in advance without causing initial data unavailability.
Solution Approach 2:
The patent introduces an intermediary mechanism - an in-memory map that stores migration state information during the recovery process. This map acts as a mediator between the incomplete metadata file scan and user I/O operations, allowing the system to provide accurate migration state information to user requests even while the background scan is still in progress, thereby maintaining data correctness without blocking user access.
2Measurement precision
If the metadata file scan is performed synchronously before allowing user I/O, then migration state accuracy is improved, but system productivity deteriorates due to suspended user operations
Solution Approach 1:
The patent segments the metadata recovery process into two independent parts: a background recovery process that scans the metadata file asynchronously, and a user I/O processing path that handles user requests with on-demand recovery. This segmentation allows the accurate migration state information to be generated in the background while user productivity is maintained through parallel processing with on-demand recovery mechanisms.
Solution Approach 2:
The system maintains continuity of useful action by implementing on-demand recovery that allows user I/O operations to continue uninterrupted during the metadata file scan. The background recovery process continues its scanning work continuously in parallel, ensuring both migration state accuracy and sustained user productivity without suspension or blocking.
3Productivity
If the metadata file is scanned in the background during restart, then user I/O availability is improved, but reliability deteriorates due to potentially unreliable migration state information
Solution Approach 1:
The in-memory map serves as a reliable intermediary that stores and provides accurate migration state information to user I/O operations. Even though the background scan may be incomplete or the metadata file potentially unreliable after non-clean shutdown, the in-memory map provides verified migration state data to user requests, ensuring reliability while maintaining continuous user I/O availability.
Solution Approach 2:
The system creates a copy of the migration state information in the in-memory map during the background recovery process. This copied information can be reliably provided to user I/O operations without requiring the user to wait for the complete metadata file scan, thus maintaining both availability and reliability by serving from the copied data structure.
Data Source
AI summary
A user I/O operation is processed in a file system of a target data storage system (DSS) performing a file system migration. The target DSS assesses a migration state of a user file of the user I/O operation, including (1) determining whether a metadata file storing migration state data for the user file has an already recovered record indicating the migration state of the file, (2) if so, then using the already recovered record to assess the migration state of the user file, and (3) otherwise (a) initiating an on-demand recovery to obtain an on-demand record for the user file, and (b) using the on-demand recovered record to assess the migration state. If the migration state indicates that the file has not yet been migrated, then the user I/O operation is redirected to a separate source DSS, and otherwise the user I/O operation is completed locally at the target DSS.