Hashgraph Erasure Coding for Consistent Data Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data centers face challenges in ensuring consistent and reliable data recovery due to inconsistencies in stored data, especially in virtual machine snapshots, and regular updates in live systems, which complicate achieving consistency in data storage and recovery.
Innovation Solution
The use of a directed acyclic graph (DAG) with Byzantine Fault Tolerance, specifically hashgraph technology, combined with erasure coding algorithms, ensures consistent data placement and recovery across storage nodes, utilizing a consensus algorithm that resists subversion unless 34% or more of the system becomes unreliable, and optimizes data restoration by generating Time Stamp Ordering and employing a gossip protocol to track available server-nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If erasure coding is applied to split data into k chunks and generate n redundant chunks, then data recovery reliability is improved, but storage space efficiency deteriorates due to file expansion by n/k times
Solution Approach 1:
The system dynamically adjusts the erasure coding parameters (k, n) based on data priority, access patterns, and storage availability. High-priority data uses higher redundancy ratios for reliability, while less critical data uses lower ratios to optimize storage efficiency. This parameter optimization resolves the contradiction by making the redundancy level adaptive rather than fixed.
Solution Approach 2:
Different storage locations and data types receive different levels of erasure coding protection. Critical data blocks are protected with higher redundancy, while non-critical blocks use lower redundancy. This localized quality approach ensures that storage space is not uniformly wasted across all data, resolving the efficiency-reliability tradeoff by applying appropriate protection only where needed.
2Productivity
If regular updates are made to live systems, then system functionality is improved, but data consistency becomes harder to achieve
Solution Approach 1:
The system performs preliminary actions by pre-calculating and staging data updates before applying them to the live system. Versioned snapshots are created and validated before activation, ensuring that updates maintain consistency even during transitions. This preliminary preparation resolves the contradiction by ensuring updates don't compromise data integrity.
Solution Approach 2:
The system implements feedback mechanisms that continuously monitor data consistency during updates. If inconsistencies are detected, the system automatically rolls back or corrects them. This feedback loop ensures that regular updates maintain data consistency, resolving the contradiction between functionality and stability.
3Ease of repair
If virtual machine snapshots are stored for recovery, then recovery capability is improved, but storage resource consumption increases
Solution Approach 1:
The system dynamically adjusts snapshot retention parameters based on virtual machine criticality, age of snapshots, and available storage. Older snapshots of less critical VMs are automatically pruned, while critical VMs retain longer snapshot histories. This parameter optimization resolves the contradiction by making snapshot retention adaptive rather than uniform.
Solution Approach 2:
The system implements intelligent snapshot management that discards redundant or obsolete snapshots while maintaining recovery capability through selective retention. Incremental snapshots are used instead of full copies, reducing storage consumption while preserving recovery ability. This resolves the contradiction by eliminating unnecessary storage overhead while maintaining essential recovery functionality.
Data Source
AI summary
A data storage system uses erasure coding in combination with hashgraph to organize stored data and recover that data in a computing environment.