Multi-Node Log Management via Dynamic Leader Failover
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional multi-node data processing systems face challenges in providing failover for leader nodes, leading to high costs due to the requirement of redundant leader nodes for each cluster, which is expensive and restrictive, and may not ensure high availability of computing resources.
Innovation Solution
Implementing a system where all leader nodes are active and can serve as failover nodes for each other, dynamically shifting management duties using listen lists and HA managers to automatically detect failures and redistribute logging and console management responsibilities without the need for redundant leader nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If redundant leader nodes are deployed for each cluster to ensure failover capability, then system reliability is improved, but device complexity and cost increase
Solution Approach 1:
Each active leader node is configured to perform multiple functions: managing its own cluster and serving as a failover node for other clusters. This multi-functionality eliminates the need for dedicated redundant leader nodes, as any leader can take over failed clusters dynamically
Solution Approach 2:
The system implements dynamic failover where leader nodes can dynamically assume management duties for failed clusters. The listen lists and HA managers enable automatic detection and redistribution of logging responsibilities without static pre-assignment of backup nodes
2Reliability
If traditional hierarchical management with dedicated backup nodes is used, then failover is ensured, but resource efficiency deteriorates due to redundant nodes
Solution Approach 1:
The failover capability is merged into the existing active leader nodes rather than being separated into dedicated backup nodes. The same leader nodes that perform primary management functions also provide failover services, consolidating resources and improving efficiency
Solution Approach 2:
The system implements self-service failover through automatic detection mechanisms (listen lists) and automated redistribution of logging duties. When a leader fails, the HA managers on remaining leaders automatically detect the failure and redistribute the failed leader's clusters without manual intervention
Data Source
AI summary
A computer-readable medium comprises instructions which, upon execution by a node in a multi-node data processing system, enable the node to serve as a first leader node by receiving system log data from multiple compute nodes in a first cluster of the multi-node data processing system, and by saving the system log data in shared storage that is also used by second and third leader nodes to save system log data for compute nodes in second and third clusters of the multi-node data processing system. The instructions further enable the node to respond to failure of either of the second and third leader nodes by automatically assuming system logging duties for the compute nodes in the cluster that was associated with the failed leader node. The instructions may also enable the node to serve as a console bridge and to save console log data in the shared storage.