Database Observability with Quorum Failover for Grey Failures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database monitoring and recovery mechanisms fail to detect partial failures or degraded performance, known as 'grey failures', leading to unexpected service interruptions and prolonged recovery times, which existing technologies like Global Data Service (GDS) from Oracle cannot address.
Innovation Solution
A database observability system that proactively identifies potential issues by using quorum-based decision logic to detect both planned and unplanned failures, including grey failures, and automatically routes traffic to a secondary instance before a complete failure occurs, utilizing a plurality of global service manager servers to monitor golden signals and trigger a failover switch when a threshold of unhealthy indications is reached.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If current monitoring mechanisms (GDS) are used to detect database failures, then hard failures can be detected and traffic routed automatically, but grey failures and degraded performance cannot be detected
Solution Approach 1:
The system segments the monitoring function by deploying multiple independent GSM servers instead of relying on a single GDS system. Each GSM server independently monitors database health using golden signals, enabling more granular detection of partial failures and degraded performance that single-point monitoring would miss.
Solution Approach 2:
The system applies partial action by monitoring only critical golden signals (CPU usage, memory usage, I/O wait, transaction rates) rather than attempting to monitor all possible database parameters. This focused monitoring approach enables detection of grey failures in specific performance dimensions while maintaining system efficiency.
2Reliability
If replicated databases are used with failover switching, then catastrophic data loss is avoided, but partial failures are hard to detect and do not trigger switching
Solution Approach 1:
The system implements continuous feedback loops where multiple GSM servers constantly query database instances and receive golden signals. This real-time feedback mechanism enables detection of partial failures and degraded performance in replicated databases, triggering failover switching when thresholds are exceeded, thereby solving the detection problem while maintaining data protection.
Solution Approach 2:
The system performs preliminary action by proactively detecting grey failures and degraded performance before they escalate to complete database failures. By monitoring golden signals and comparing them against thresholds, the system can initiate failover switching in advance, preventing catastrophic data loss while maintaining service continuity.
3Measurement precision
If multiple GSM servers are deployed to monitor golden signals, then detection accuracy improves, but system complexity increases
Solution Approach 1:
The system merges the monitoring efforts of multiple independent GSM servers through a coordinated architecture where each server monitors the same golden signals across database instances. By combining their observations and applying quorum-based logic, the system achieves high detection accuracy while managing complexity through standardized monitoring protocols and shared threshold configurations.
Solution Approach 2:
Each GSM server is designed as a universal monitoring component that can monitor any database instance using standardized golden signals. This multi-functional design allows the same monitoring logic to be applied across multiple database instances and replicated pairs, reducing overall system complexity through reuse of proven monitoring patterns rather than custom monitoring for each instance.
Data Source
AI summary
A computerized method is provided for managing replicated database resources. Systems and methods described can use a plurality of geographically distributed servers to query a replicated database and determine the databases health based on monitored golden signals in response to the query. The various servers can be geographically distributed and can report their database health findings to a monitoring agent which can use quorum logic based on all of the reporting servers to identify resource health and trigger failover to another instance of the replicated database when warranted. Such systems and methods can thereby detect not only hard failures but also so-called grey failures resulting in diminished resource performance and trigger failovers to increase resource performance.