Database Failure Detection via Heartbeat Table and Disk Write Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for determining master database failure in a relational database cluster are inaccurate and difficult to apply in complex scenarios, often misinterpreting failures due to port checks or request monitoring, which affects Recovery Time Objective (RTO).
Innovation Solution
A method and apparatus that perform write and read operations on a heartbeat table, write to a hard disk, and analyze connection thresholds and master-slave synchronization status to accurately detect database failures, outputting specific failure information and potentially switching data to a candidate database.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If port check method is used to determine database status, then the detection method is simple, but false positives occur when machine is suspended or disk fails while port remains alive
Solution Approach 1:
The detection method is segmented into multiple independent detection dimensions: connection detection, heartbeat table write operation detection, read operation detection, and hard disk write operation detection. Each dimension independently checks a specific aspect of database functionality, allowing comprehensive failure detection without requiring a single complex detection mechanism.
Solution Approach 2:
The heartbeat table serves as an intermediary object to detect database functionality. By attempting to write to and read from the heartbeat table, the system indirectly assesses whether the database can perform critical operations without directly querying the database status, thus avoiding false positives from port-level checks.
2Measurement precision
If request monitoring method is used to determine database status, then failure detection becomes more accurate, but the database is easily misinterpreted as abnormal when under large pressure
Solution Approach 1:
The detection mechanism dynamically adapts to different database states by implementing a multi-stage detection process. The system first checks connections, then attempts heartbeat table operations, and finally performs hard disk write operations. This dynamic progression allows the system to differentiate between temporary pressure-induced slowdowns and actual failures, improving adaptability to complex scenarios.
Solution Approach 2:
The system performs preliminary detection actions in a structured sequence before concluding database failure. By first checking connections, then attempting lightweight heartbeat table operations, and only concluding failure after confirming inability to perform hard disk writes, the system avoids premature failure declarations during high-pressure periods.
3Ease of manufacture
If simple determination approaches are used, then the detection method is easy to implement, but they are difficult to be applied to actually complicated application scenarios
Solution Approach 1:
The complex detection task is segmented into four manageable, independently implementable detection modules: connection detection, heartbeat table write detection, heartbeat table read detection, and hard disk write detection. Each module can be implemented and maintained separately, preserving ease of implementation while collectively providing comprehensive coverage for complex application scenarios.
Solution Approach 2:
The detection system is designed with universal applicability across different database types and failure scenarios. The multi-dimensional detection approach (connection, read operations, write operations, hard disk operations) can detect various failure modes including machine suspension, database instance suspension, disk failures, and network issues, making the system versatile without requiring scenario-specific customization.
Data Source
AI summary
Embodiments of the present disclosure disclose a method and apparatus for outputting information. A specific embodiment of the method comprises: connecting a database, in response to receiving a request for detecting the database; performing a write operation on a heartbeat table in the database, in response to detecting the connection with the database being normal; performing a read operation on the heartbeat table in the database, in response to detecting the write operation for the heartbeat table being abnormal; performing on the database an operation of writing a file to a hard disk, in response to detecting the read operation on the heartbeat table being normal; and outputting first information for representing a cause of a database failure, in response to detecting the operation of writing the file to the hard disk being abnormal. This embodiment improves the accuracy of the detection on the database failure.