Scalable Storage Protection via Local and Global Redundancy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Scalable storage systems face challenges in balancing redundancy for data protection across multiple nodes while minimizing data shipping, latency, and redundancy costs, as existing solutions like erasure-coding require excessive data transmission and computational resources.
Innovation Solution
A storage system with local and global redundancy mechanisms, where each processing node computes and stores local and delta data, and shares delta data with other nodes to determine global redundant data, reducing the amount of data shipped between nodes and optimizing redundancy for intra-node and inter-node failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If erasure-coding techniques are used to provide protection by distributing error-correction coding over a larger number of disks, then data protection reliability is improved, but the amount of data that must be shipped between nodes increases significantly
Solution Approach 1:
The patent segments redundancy protection into two distinct layers: local redundancy within each node and global redundancy across nodes. Each node independently computes local redundant data for its own disks, and only ships minimal delta data (differences between old and new data) to other nodes for global redundancy. This segmentation reduces the volume of data shipped compared to traditional erasure-coding that requires shipping entire data blocks across the network.
Solution Approach 2:
The patent applies local quality by implementing node-specific redundancy computation where each processing node independently computes local redundant data based on its own stored data. This local computation eliminates the need to ship large amounts of data across the network, as each node maintains its own redundancy locally while sharing only minimal delta information globally.
2Reliability
If data is updated at r separate disks to handle r out of n drive failures, then fault tolerance is improved, but the amount of redundancy required and data shipping increases
Solution Approach 1:
The patent divides fault tolerance into two segmented layers: local fault tolerance handled by intra-node redundancy and global fault tolerance handled by inter-node redundancy. This segmentation allows the system to achieve r out of n drive failure protection without requiring redundancy across all nodes, thereby reducing overall redundancy overhead and complexity.
Solution Approach 2:
The patent implements partial redundancy by having each node compute only the portion of redundant data needed for local protection, and ship only delta data (partial updates) to other nodes rather than complete redundant copies. This partial action approach achieves the required fault tolerance while minimizing redundancy overhead.
3Reliability
If large amounts of data are shipped between nodes for updating or recovery, then global data consistency is improved, but latency and network bandwidth consumption increase
Solution Approach 1:
The patent extracts only the essential information needed for global redundancy by computing and shipping delta data (differences between old and new data) rather than shipping entire data blocks. This extraction approach maintains global data consistency while dramatically reducing the volume of data shipped and the associated latency.
Solution Approach 2:
The patent changes the parameter being transmitted from complete data blocks to delta data (differences). This parameter transformation reduces the amount of data that must be shipped across the network while still achieving global data consistency, thereby reducing latency and network bandwidth consumption.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The disclosure is directed to protecting data of a scalable storage system. A scalable storage system includes a plurality of nodes, each of the nodes having directly-attached storage (DAS), such as one or more hard-disk drives and/or solid-state disk drives. The nodes are coupled via an inter-node communication network, and a substantial entirety of the DAS is globally accessible by each of the nodes. The DAS is protected utilizing intra-node protection to keep data stored in the DAS reliable and globally accessible in presence of a failure within one of the nodes. The DAS is further protected utilizing inter-node protection to keep data stored in the DAS reliable and globally accessible if at least one of the nodes fails.