Erasure Code Parameter Selection for Repair-Free Data Retention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for predicting the reliability of erasure-coded data require costly repair processes, which involve significant computational overhead and network traffic, and are not always necessary, as not all data needs to be kept indefinitely.
Innovation Solution
A new data storage technique using a continuous-time Markov chain model that calculates the Mean Time to Data Loss (MTTDL) without the need for repair, allowing for sufficient erasure coding redundancy to maintain data reliability for a prescribed period by varying erasure coding parameters N and K.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data repair is performed to maintain reliability, then data reliability is improved, but computational overhead and network traffic increase significantly
Solution Approach 1:
The patent applies preliminary action by pre-calculating and storing the Mean Time to Data Loss (MTTDL) values for different erasure coding parameters before actual data storage. When storing data, the system simply queries the pre-computed MTTDL table to determine appropriate parameters, avoiding the need for real-time reliability calculations and repair operations. This eliminates computational overhead during data storage while maintaining reliability through informed parameter selection.
Solution Approach 2:
The patent replaces the mechanical repair process with a mathematical modeling approach. Instead of physically repairing damaged data through complex erasure coding algorithms and network operations, the system uses a Markov chain model to predict reliability outcomes beforehand. This substitution transforms an active mechanical repair system into a passive predictive modeling system, eliminating computational overhead during operation.
2Reliability
If data repair is performed to maintain reliability, then data reliability is improved, but network traffic increases significantly
Solution Approach 1:
The system performs preliminary action by pre-computing MTTDL values for various erasure coding configurations and storing them in a lookup table. During data storage operations, the system queries this pre-computed table to select optimal parameters without triggering any network traffic. This eliminates the need for real-time communication with other nodes that would otherwise be required for repair operations.
Solution Approach 2:
The patent introduces an intermediary element in the form of a pre-computed MTTDL table that mediates between data storage requirements and reliability goals. Instead of directly performing repair operations that generate network traffic, the system uses this intermediary table to determine optimal parameters in advance, thereby eliminating network traffic during actual data storage while maintaining reliability.
3Reliability
If erasure coding parameters N and K are increased to improve reliability, then data reliability is improved, but storage overhead increases
Solution Approach 1:
The patent applies parameter changes by systematically varying the erasure coding parameters N and K in the Markov chain model to compute MTTDL values for different configurations. The pre-computed MTTDL table stores results for multiple parameter combinations, allowing the system to select the optimal balance between reliability and storage overhead based on specific requirements, rather than using fixed parameters.
Solution Approach 2:
The system applies partial action by allowing users to specify minimum reliability requirements, and then selecting the smallest N and K values from the pre-computed MTTDL table that satisfy those requirements. This avoids excessive redundancy by choosing the minimal necessary erasure coding parameters rather than always using maximum redundancy, thereby reducing storage overhead while meeting reliability goals.
Data Source
AI summary
An improved data storage technique achieves a desired level of reliability by providing sufficient redundancy in erasure coded data to maintain the data, without repair, for a prescribed period of time. The improved technique employs a newly devised, continuous-time Markov chain model. The model can be applied in computerized systems to establish erasure coding parameters for storing and reliably maintaining data for a designated period of time, without any need to repair the data to reestablish an original or previous level of erasure coding redundancy.