Erasure Coding Sequencing for Consistent Distributed Data Stores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing erasure coding methods struggle to maintain data consistency and correctness across multiple data stores, especially during constant updates and concurrent failures, leading to potential data loss and incorrect reconstruction.
Innovation Solution
A sequencing method is applied to assign sequence numbers to data and checksum blocks, enabling efficient erasure encoding and decoding, and a recovery process to identify and fix inconsistencies, ensuring data consistency and self-healing across distributed data stores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If algebraic-based erasure coding is applied to generate M redundant data blocks from N original data blocks, then data reliability and failure resiliency are improved, but data consistency management complexity increases and the system becomes vulnerable to inconsistency during update failures
Solution Approach 1:
The patent applies preliminary action by pre-generating sequence numbers for all data blocks before the encoding process. Each data block is assigned a sequence number that reflects its version state. This preliminary numbering system enables the system to track and identify consistency states without requiring complex real-time coordination during encoding operations, thus resolving the contradiction between reliability and management complexity
Solution Approach 2:
The patent implements feedback mechanisms where the system continuously monitors the consistency state of data blocks by comparing sequence numbers. When inconsistencies are detected (such as when some redundant blocks are updated while others are not), the system provides feedback to trigger recovery operations. This feedback loop maintains data reliability while keeping consistency management tractable through automated detection and correction
2Manufacturing precision
If erasure encoding is executed to update all M redundant data blocks when a data change is detected, then data correctness is improved, but the system becomes vulnerable to concurrent failures during the update process
Solution Approach 1:
The patent applies preliminary action by assigning sequence numbers to data blocks before the encoding update process begins. This pre-numbering system allows the encoding operation to proceed without requiring complex real-time consistency checks, reducing the window of vulnerability to concurrent failures while maintaining data correctness through post-update sequence number verification
Solution Approach 2:
The patent implements beforehand cushioning by using sequence numbers as a protective mechanism. Before encoding updates are applied, the system establishes a baseline sequence number state. If concurrent failures occur during the update process, this pre-established sequence information serves as a cushion that allows the system to detect and recover from inconsistencies, thus protecting against the vulnerability introduced by executing encoding during potential failure windows
3Stability of the object's composition
If re-encoding is performed to regenerate all redundant data blocks when inconsistency is detected, then data consistency is restored, but productivity decreases due to unnecessary re-encoding operations
Solution Approach 1:
The patent applies partial action by using sequence numbers to identify only the specific data blocks that are inconsistent, rather than re-encoding all redundant blocks. The recovery process targets only the affected blocks based on sequence number mismatches, restoring data consistency while minimizing unnecessary re-encoding operations and preserving system productivity
Solution Approach 2:
The patent implements parameter changes by using sequence numbers as a metadata parameter that enables efficient consistency verification. Instead of expensive full re-encoding operations, the system changes the approach to use lightweight sequence number comparisons to detect inconsistencies, then applies targeted re-encoding only when and where needed, thus maintaining consistency while avoiding productivity loss from excessive operations
4Measurement precision
If sequence numbers are assigned to all data and checksum blocks, then data consistency identification is improved, but device complexity increases due to additional metadata management
Solution Approach 1:
The patent applies universality by designing the sequence number system to serve multiple functions simultaneously: it acts as a version identifier, a consistency marker, and a recovery guide. This multi-functional approach eliminates the need for separate metadata structures for each purpose, improving consistency identification precision while avoiding the complexity increase that would result from multiple separate tracking mechanisms
Solution Approach 2:
The patent implements homogeneity by applying the same sequence number structure uniformly to all data blocks and checksum blocks across the distributed system. This homogeneous approach simplifies metadata management by using a consistent format and interpretation rule everywhere, enabling precise consistency identification without the complexity of handling different metadata types for different block kinds
Data Source
AI summary
An efficient method to apply an erasure encoding and decoding scheme across dispersed data stores that receive constant updates. A data store is a persistent memory for storing a data block. Such data stores include, without limitation, a group of disks, a group of disk arrays, or the like. An encoding process applies a sequencing method to assign a sequence number to each data and checksum block as they are modified and updated onto their data stores. The method preferably uses the sequence number to identify data set consistency. The sequencing method allows for self-healing of each individual data store, and it maintains data consistency and correctness within a data block and among a group of data blocks. The inventive technique can be applied on many forms of distributed persistent data stores to provide failure resiliency and to maintain data consistency and correctness.