Locally Generated Erasure Codes for Data Center Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data redundancy in data centers leads to increased network resource consumption and performance degradation due to the time-consuming and costly communication of large data sets across networks, as well as computationally expensive redundancy mechanisms.
Innovation Solution
Implementing locally generated simple erasure codes, such as XOR functionality, to maintain redundant copies of data in an encoded state, which reduces storage capacity requirements and allows for efficient generation and decoding of data, thereby minimizing network resource consumption and enabling faster recovery from failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If redundant copies of data are maintained across multiple failure domains or data centers, then data reliability is improved, but network resource consumption increases and performance degrades
Solution Approach 1:
The patent segments the redundant data storage into encoded portions distributed across multiple failure domains. Instead of storing complete duplicate copies, the data is divided and encoded such that any subset of encoded copies can reconstruct the original data, reducing the amount of data transmitted over the network while maintaining reliability across distributed locations.
Solution Approach 2:
The patent transforms the redundancy mechanism by applying erasure coding algorithms that change the parameter representation of data. Rather than storing identical copies, the system encodes data into a different mathematical representation where k data blocks can be transformed into n encoded blocks (where n > k), allowing reconstruction with fewer network resources consumed during redundancy operations.
2Reliability
If traditional redundancy mechanisms are implemented, then data protection is improved, but processing capability utilization increases due to computationally expensive operations
Solution Approach 1:
The patent replaces complex mechanical redundancy operations with simpler mathematical encoding operations. By substituting traditional copy-based redundancy with erasure coding that uses XOR operations and polynomial-based encoding, the system achieves equivalent or superior data protection with significantly reduced computational overhead during both encoding and decoding phases.
Solution Approach 2:
The patent changes the computational parameters of redundancy by using efficient erasure coding algorithms that operate in linear time complexity. The encoding process transforms k data blocks into n encoded blocks using systematic mathematical operations that are computationally lighter than traditional replication and verification mechanisms, reducing processing capability utilization while maintaining robust data protection.
3Reliability
If large data sets are communicated across networks for redundancy, then data availability is improved, but communication time increases
Solution Approach 1:
The patent extracts only the essential redundancy information needed for data protection and transmits it separately from the main data sets. By separating the redundancy encoding from the bulk data communication, the system maintains data availability through distributed encoded copies while minimizing the communication time required for redundancy operations, as only encoded portions need to be transmitted across the network.
Data Source
AI summary
An “erasure code” is an encoding of multiple different sets of data. Redundant copies of data are maintained in such erasure codes, thereby utilizing only a fraction of the storage capacity of unencoded copies. Erasure codes are efficiently generated, with a minimum of processing resources utilizing XOR functionality. Additionally, erasure codes are generated from local data, thereby avoiding the consumption of network resources. At least one unencoded copy of a set of data is maintained, while the remaining, redundant copies are encoded into erasure codes. Requests for data are provided from the unencoded copy. Should it fail, a new unencoded copy can be generated by another computing device having access to both an erasure code as well as unencoded copies of the other data that was also pressed into that erasure code. Multiple failures can be survived through recursive application of such a decoding of encoded data.