Cluster Failure Notification and Cache Flush Mechanism
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large computer systems face challenges in recovering from catastrophic failures, such as system-wide power failures or natural disasters, due to delayed and incomplete data recovery processes, especially when relying on node-by-node or application-by-application data persistence.
Innovation Solution
Implementing a system where nodes in a multi-node cluster can send failure notifications to freeze processes and flush non-persistent data to non-volatile memory system-wide, enabling a swift and comprehensive data preservation and recovery mechanism.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If node-by-node or application-by-application data persistence is used, then system complexity is reduced and ease of operation is improved, but data recovery completeness and reliability deteriorate during catastrophic failures
Solution Approach 1:
The system performs preliminary actions by freezing all processes and flushing caches to non-volatile memory immediately upon detecting a failure notification, before data loss occurs. This proactive approach ensures data is preserved in a safe state before attempting recovery, resolving the contradiction by prioritizing reliability through pre-established protection mechanisms.
Solution Approach 2:
The system segments the failure response into distinct phases: detection (receiving failure notification), immediate action (freezing processes and flushing caches), and recovery (restoring from non-volatile memory). This segmentation allows each phase to be optimized independently, maintaining operational simplicity while ensuring complete data recovery through systematic handling of each recovery stage.
2Reliability
If system-wide failure notifications are implemented, then data recovery reliability is improved, but communication overhead and system complexity increase
Solution Approach 1:
The system merges individual node failure detections into a unified system-wide failure notification mechanism. When any node detects a failure, it triggers a global response that coordinates all nodes to freeze processes and flush caches simultaneously. This merging approach improves reliability by ensuring consistent system-wide protection while managing communication overhead through a centralized notification protocol.
Solution Approach 2:
The system introduces a failure notification mechanism as an intermediary that mediates between individual node failures and system-wide recovery actions. This intermediary receives failure notifications from any node and coordinates the appropriate response, reducing direct communication overhead between nodes while maintaining reliable system-wide data protection through centralized coordination.
3Reliability
If processes are frozen and caches flushed to non-volatile memory upon failure detection, then data loss risk is reduced, but system productivity and speed deteriorate during failure events
Solution Approach 1:
The system applies preliminary anti-action by freezing processes and flushing caches to non-volatile memory immediately upon detecting a failure notification, counteracting the potential harm of data loss before it occurs. This protective action prioritizes data integrity over continued productivity during failure events, ensuring that data preservation takes precedence when system reliability is threatened.
Solution Approach 2:
The system provides beforehand cushioning by maintaining non-volatile memory as a protective buffer that can absorb data loss during catastrophic failures. By continuously syncing critical data to this cushioning layer and freezing processes upon failure detection, the system creates a safety margin that protects against total data loss, accepting temporary productivity reduction as necessary protection against irreversible data loss.
Data Source
AI summary
Examples include a system comprising a non-volatile memory, a cluster management interface, and a multi-node cluster. In some examples, the cluster management interface may monitor a system critical alert to determine if the system critical alert has been triggered. Based on the determination that it has been triggered, the cluster management interface may multicast a system failure notification. The multi-node cluster of the system has multiple nodes, each node connected to the non-volatile memory and having a processor and a processor cache. Each node of the multi-node cluster may determine if the system failure notification has been received and based on the determination that it has been received, each node may freeze execution of all processes on the process and flush the processor cache to the non-volatile memory.