Sub-health OSD Identification in Distributed Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional distributed storage systems face significant delays in monitoring and processing sub-health faults in storage nodes, leading to prolonged system unavailability and compromised availability.
Innovation Solution
A method and system for identifying and managing sub-health Object Storage Devices (OSDs) through a distributed storage controller that utilizes a management node to receive health status reports from OSDs, determine sub-health states, and dynamically reconfigure partition allocation views to ensure data redundancy and availability by identifying and isolating affected nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional heartbeat monitoring is used to detect sub-health faults, then the system can identify offline failures, but the detection delay is too long and availability is compromised
Solution Approach 1:
The patent implements preliminary action by having OSDs proactively report their health status at regular intervals before complete failure occurs. The management node collects these pre-failure health reports and can identify sub-health states early, enabling timely intervention before the OSD goes completely offline. This transforms the reactive heartbeat model into a proactive health monitoring system.
Solution Approach 2:
The patent establishes a feedback mechanism where OSDs continuously report health status to the management node, which then analyzes the information and provides feedback by identifying sub-health states and triggering appropriate responses. This closed-loop feedback system enables real-time monitoring and rapid response to deteriorating OSD conditions, significantly reducing detection delay compared to conventional methods.
2Reliability
If offline isolation processing is performed on sub-health nodes, then fault impact is contained, but system availability is greatly compromised during the isolation process
Solution Approach 1:
The patent applies dynamics by implementing online isolation with dynamic data redirection. Instead of static offline isolation that takes the node completely out of service, the system dynamically redirects I/O operations from sub-health OSDs to healthy OSDs in real-time. This dynamic adaptation maintains system availability while still isolating the faulty node, allowing continuous operation without service interruption.
Solution Approach 2:
The management node serves as an intermediary that coordinates the isolation process. It receives health status reports, identifies sub-health OSDs, and orchestrates the redirection of data operations to healthy OSDs. This intermediary coordination enables seamless online isolation where the faulty node is gradually taken out of service while maintaining system functionality, rather than causing abrupt availability loss.
3Loss of time
If health status information is collected and analyzed in real-time, then sub-health OSDs can be identified timely, but the system complexity increases
Solution Approach 1:
The management node is designed with multi-functionality, serving as a universal component that performs health status collection, analysis, decision-making, and coordination of isolation operations. By consolidating these functions in a single multi-functional node rather than distributing complex monitoring logic across all OSDs, the system achieves real-time identification capability without proportionally increasing overall system complexity.
Solution Approach 2:
OSDs autonomously generate and report their own health status information without requiring external probing or complex monitoring infrastructure. Each OSD self-evaluates its health state and reports it to the management node, eliminating the need for complex active monitoring mechanisms. This self-service approach enables timely identification with minimal added system complexity.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments of the present invention relate to a method for identifying a sub-health OSD, an apparatus, and a system. A primary OSD that processes a read/write data request and that is in a data storage system reports, to a management node in the system, a delay in processing the read/write data request by a secondary OSD. The management node determines, based on a received report message, which OSDs in the system are in a sub-health state. Because collected data is more comprehensive, the management node can determine a sub-health OSD in the system more accurately.