Erasure Coding Metadata Distribution Across Fault Domains
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing methods for managing data in computing devices, particularly in data clusters, face challenges in efficiently utilizing computing resources for data generation, storage, and backup, leading to performance impacts due to the lack of effective distribution and redundancy strategies.
Innovation Solution
A method that involves applying erasure coding to data from a host to generate data chunks and parity chunks, followed by deduplication and storage across multiple fault domains, with storage metadata distributed across both accelerator and non-accelerator pools to ensure data redundancy and availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored with full redundancy across multiple fault domains, then data availability and reliability are improved, but storage resource consumption increases
Solution Approach 1:
The patent segments data into data chunks and applies erasure coding to generate parity chunks, distributing them across multiple fault domains. This segmentation allows the system to achieve redundancy without storing complete copies of all data, thereby improving reliability while reducing storage consumption compared to full replication.
Solution Approach 2:
The patent changes the storage parameter from full replication to erasure coding with deduplication. By transforming the redundancy mechanism and applying deduplication before erasure coding, the system achieves the same reliability level with reduced storage resource consumption, as identical data blocks are stored only once.
2Reliability
If multiple copies of data are stored for backup, then data recovery capability is improved, but computing resource utilization deteriorates
Solution Approach 1:
The patent uses erasure coding to generate parity chunks that can reconstruct original data, rather than storing multiple complete copies. This approach provides data recovery capability while consuming fewer computing resources for storage operations, as parity chunks are computed efficiently and stored once per fault domain.
3Productivity
If storage metadata is centralized in accelerator pool, then data management efficiency is improved, but system reliability deteriorates due to single point of failure
Solution Approach 1:
The patent segments storage metadata distribution by storing it in both accelerator pool and non-accelerator pool across multiple fault domains. This segmentation eliminates the single point of failure while maintaining data management efficiency, as metadata is distributed redundantly across different storage locations.
4Quantity of substance
If erasure coding and deduplication are applied to all data, then storage efficiency is improved, but processing time increases
Solution Approach 1:
The patent applies deduplication before erasure coding, performing preliminary data processing to eliminate redundant blocks. This preliminary action reduces the amount of data that subsequently requires erasure coding operations, thereby improving storage efficiency while minimizing the additional processing time that would result from applying both operations to the full dataset.
Data Source
AI summary
In general, the invention relates to a method for managing data, the method includes obtaining data from a host, applying an erasure coding procedure to the data to obtain a plurality of data chunks and at least one parity chunk, deduplicating the plurality of data chunks to obtain a plurality of deduplicated data chunks, generating storage metadata associated with the plurality of deduplicated data chunks and the at least one parity chunk, storing the storage metadata in an accelerator pool, storing, across a plurality of fault domains, the plurality of deduplicated data chunks and the at least one parity chunk, and initiating storage metadata distribution on the storage metadata across the plurality of fault domains.