Inline Deduplication with Erasure Coding for Low-Overhead Backup
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The process of generating, storing, and backing up data in computing devices consumes significant computing resources, impacting overall performance due to inefficient data management and storage operations.
Innovation Solution
A method involving erasure coding and deduplication is applied to data, dividing it into chunks and parity chunks, which are then stored across multiple nodes in a data cluster, minimizing read and write operations by tracking location information, thereby optimizing storage efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional data backup methods are used, then data can be stored and backed up, but computing resources are significantly consumed and overall performance is impacted
Solution Approach 1:
The patent segments data into fixed-size chunks before applying erasure coding. This segmentation allows the system to process and store data in manageable units, enabling parallel processing across multiple nodes while reducing the computational overhead of handling large data blocks as single units.
Solution Approach 2:
The patent transforms data from its original form into encoded form by changing parameters through erasure coding operations. This parameter transformation converts k data chunks into n encoded chunks (where n > k), creating redundancy that improves reliability while the distributed storage across nodes optimizes resource utilization.
2Reliability
If erasure coding is applied to all data chunks, then data redundancy and availability are improved, but the number of read and write operations increases
Solution Approach 1:
The patent performs deduplication on data chunks before applying erasure coding. This preliminary action identifies and eliminates duplicate chunks, reducing the total number of chunks that require encoding and storage. By removing redundancy beforehand, the system minimizes the subsequent read and write operations needed while still achieving the desired data availability through erasure coding of the remaining unique chunks.
3Reliability
If data is distributed across multiple nodes, then fault tolerance is improved, but system complexity increases
Solution Approach 1:
The patent implements a universal metadata structure that serves multiple functions: tracking chunk locations across nodes, monitoring data availability, and managing fault recovery. This multi-functional metadata system simplifies the overall architecture by consolidating multiple management tasks into a single coherent framework, reducing system complexity while maintaining fault tolerance across distributed nodes.
Data Source
AI summary
A method for storing data includes obtaining data, applying an erasure coding procedure to the data to obtain a plurality of data chunks and a parity chunk, deduplicating the plurality of data chunks to obtain a plurality of deduplicated data chunks, storing, across a plurality of nodes, the plurality of deduplicated data chunks and the parity chunk, and tracking location information for each of the plurality of deduplicated data chunks and the parity chunk.