Distributed NVMe Drive Recovery via Controller Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional recovery mechanisms for previous drive technologies, such as Serial Attached SCSI (SAS) drives, are not easily applicable to multi-controller NVMe drives due to differences in controller functionality and the complexity of managing failures in large-scale NVMe storage systems, where each controller is an independently functioning unit.
Innovation Solution
A distributed management system for multi-controller NVMe drives that detects path failures, attempts controller-level resets, and if unsuccessful, coordinates a sub-system level reset by electing a master storage node to quiesce and shut down controllers, followed by a drive-level reset, utilizing timeouts to constrain operation completion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If controller-level resets are used for recovery, then recovery speed is improved and other controllers remain operational, but the solution becomes complex due to the need for distributed coordination among multiple storage nodes
Solution Approach 1:
The patent segments the NVMe drive into multiple independent controllers, each capable of independent reset operations. This allows selective recovery of individual controllers without affecting the entire drive, enabling faster recovery by targeting only the failed controller rather than performing a complete drive reset.
Solution Approach 2:
Each storage node is empowered to independently detect path failures and initiate controller-level reset operations for its associated controllers. This distributed self-service approach eliminates the need for centralized coordination, reducing system complexity while maintaining fast recovery capabilities.
2Reliability
If sub-system level resets are used for recovery, then reliability is improved by ensuring complete system reset, but productivity deteriorates due to extended downtime and loss of all controllers
Solution Approach 1:
The patent divides the reset operation into controller-level segments, allowing the system to reset only the specific failed controller rather than the entire NVMe drive. This segmentation maintains storage availability by keeping other controllers operational while still achieving reliable recovery of the failed component.
Solution Approach 2:
The patent applies different recovery actions to different controllers based on their operational status. Controllers that are functioning normally continue to operate without interruption, while only the failed controller undergoes reset. This local quality approach ensures high productivity by maintaining service from healthy controllers while recovering the failed one.
3Ease of operation
If conventional SAS recovery mechanisms are applied to NVMe drives, then ease of operation is maintained through familiar procedures, but adaptability deteriorates due to fundamental differences in controller architecture and independence
Solution Approach 1:
The patent implements a dynamic recovery mechanism that adapts to the specific failure mode and controller architecture of NVMe drives. Unlike static SAS procedures, the system can selectively apply controller-level resets or coordinate sub-system level resets based on the detected failure type, making it adaptable to the independent controller architecture of NVMe while maintaining operational simplicity through automated detection and response.
Data Source
AI summary
A method of distributed management of recovery of multi-controller NVMe drives includes detecting a path failure of a PCIe path from a first storage node to a first controller on the multi-controller NVMe drive, and initially attempting to correct the path failure using a controller level reset. If the controller level reset is unsuccessful, an alternative path to the controller is sought, and if that is unsuccessful a drive level reset operation is coordinated by all storage nodes with controllers executing on the NVMe drive. To coordinate reset of the NVMe drive, one storage node is elected master. Each node (both slave and master) initiates quiescing of IO operations on its respective controller, and after quiescing has completed, initiates shutdown of its respective controller. Once all controllers are shut down, the master initiates reset of the NVMe drive. Timeouts are used to constrain completion of the quiescing and shutdown operations.