Scale-out Storage Metadata Redundancy for Failover Resilience
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Scale-out storage systems face challenges in properly protecting metadata, which is crucial for system control and failover processes, due to the lack of effective redundancy and data persistence mechanisms.
Innovation Solution
A scale-out storage system is designed with multiple computer nodes forming redundancy groups, where metadata is redundantly stored across both node memories and permanent storage devices, enabling failover and ensuring data persistence through redundant storage across different nodes and storage devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If metadata is stored only in node memory for fast access, then control speed is improved, but data persistence and protection against node failure deteriorates
Solution Approach 1:
The system performs preliminary actions by redundantly storing metadata in permanent storage devices before potential failures occur. Each node maintains metadata in its memory for fast access while simultaneously storing redundant copies in permanent storage, ensuring both speed and reliability are achieved.
Solution Approach 2:
The system creates redundant copies of metadata stored in permanent storage devices across multiple nodes. When a node fails, the metadata can be recovered from the permanent storage or retrieved from other nodes in the redundancy group, ensuring data persistence without sacrificing access speed during normal operation.
2Reliability
If metadata is redundantly stored in permanent storage devices, then data persistence is improved, but access speed deteriorates
Solution Approach 1:
The system performs preliminary action by pre-loading metadata from permanent storage into node memory before it is needed for control operations. This ensures that during normal operation, metadata can be accessed quickly from memory while maintaining redundant copies in permanent storage for persistence.
Solution Approach 2:
Node memory acts as an intermediary between permanent storage devices and the control system. Metadata is stored in permanent storage for persistence, then quickly loaded into node memory for fast access during control operations, eliminating the need for slow direct access to permanent storage during normal operations.
3Device complexity
If metadata is stored in a single primary node, then system complexity is reduced, but failover capability deteriorates
Solution Approach 1:
The system segments metadata management by dividing nodes into redundancy groups with primary and secondary roles. Each group independently manages its own metadata, allowing failover within the group without affecting the entire system. This segmentation provides failover capability while keeping individual group complexities manageable.
Solution Approach 2:
The system creates redundant copies of metadata across primary and secondary nodes within each redundancy group. When a primary node fails, the secondary node with the copied metadata can take over, providing failover capability. The modular redundancy group structure keeps system complexity localized rather than system-wide.
Data Source
AI summary
A scale-out storage system includes a plurality of computer nodes each of which has a memory and a processor, and a storage apparatus. The computer nodes have one or more redundancy groups each of which is a group for metadata protection. Each of the one or more redundancy groups includes two or more of the computer nodes including a primary node being a primary computer node and a secondary node being a secondary computer node, and a failover is performed from the primary node to the secondary node. The memory of the primary node has stored therein metadata related to the redundancy group and to be accessed for control. The metadata is redundantly stored in the memory of the primary node and the memory of the secondary node.