Inline Deduplication with Erasure Coding for Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing methods for managing data in computing devices, such as generating, storing, and backing up data, lead to inefficient utilization of computing resources, impacting overall performance due to the resource-intensive nature of these processes.
Innovation Solution
A method involving erasure coding and deduplication is applied to data, dividing it into chunks and parity chunks, which are then deduplicated and stored across multiple nodes in a data cluster, utilizing a heterogeneous pool of nodes with varying computing resources to optimize storage efficiency and availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional data backup methods are used, then data redundancy is achieved, but storage efficiency deteriorates due to duplicate data copies
Solution Approach 1:
The patent segments data into fixed-size chunks before applying erasure coding. This segmentation allows the deduplication process to work at the chunk level, identifying and eliminating duplicate chunks while maintaining the ability to reconstruct complete files from remaining chunks. The segmentation transforms the problem from storing entire duplicate files to storing only unique data chunks.
Solution Approach 2:
The patent changes the parameter of data representation by applying erasure coding transformations. Instead of storing raw duplicate data, the system transforms data into encoded chunks where redundancy is mathematically embedded. This parameter change allows the system to store fewer physical copies while maintaining the same level of data availability through cryptographic reconstruction.
2Reliability
If comprehensive data processing and storage operations are performed, then data security and availability are improved, but computing resource utilization deteriorates
Solution Approach 1:
The patent performs deduplication operations on data chunks before the actual storage operation. By identifying and removing duplicate chunks in advance, the system reduces the volume of data that needs to be transmitted and stored, thereby reducing the computational load during the main storage process while maintaining data security through the erasure coding framework.
3Reliability
If data is stored across multiple nodes, then fault tolerance is improved, but system complexity deteriorates
Solution Approach 1:
The patent creates a universal data chunk format that can be stored across any node in the distributed system. Each chunk is self-contained with metadata that allows it to be independently managed, transmitted, and reconstructed. This universality simplifies the system architecture by allowing any node to perform any function (storage, retrieval, reconstruction) without requiring specialized configurations.
Data Source
AI summary
A method includes obtaining a data, applying an erasure coding procedure to the data to obtain a plurality of data chunks and at least one parity chunk, deduplicating the plurality of data chunks to obtain a plurality of deduplicated data chunks, and storing, across a plurality of nodes, the plurality of deduplicated data chunks and the at least one parity chunk.