Deduplicated Erasure Coding for Chunk Lookup Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage approaches face challenges in efficiently managing the chunk-lookup disk bottleneck problem and balancing reliability and efficiency, particularly when combining deduplication and erasure coding, which often result in sub-optimal performance and resource utilization.
Innovation Solution
The implementation of joint de-duplication and erasure coding using systematic fountain codes and unequal error protection (UEP) erasure coding, which involves inline chunk-based de-duplication, weak hashing, and multiple precoding stages to reduce hash collisions and memory footprint, while providing flexible and reliable data storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If strong hash functions are used to identify unique chunks, then hash collision rate is reduced, but memory consumption increases
Solution Approach 1:
The patent divides the hash identification process into multiple stages: first using a weak hash function to generate a preliminary chunk ID, then using a strong hash function only on unique chunks to generate final unique IDs. This segmentation reduces the number of strong hash computations and memory storage requirements while maintaining low collision rates.
Solution Approach 2:
The patent applies different hash function strengths to different data characteristics: weak hash functions are used for initial filtering and duplicate detection, while strong hash functions are applied only when needed for final unique identification. This local differentiation optimizes the balance between collision resistance and resource consumption.
2Speed
If chunk ID table is stored in RAM for fast access, then lookup speed is improved, but device complexity increases when paging to disk is required
Solution Approach 1:
The patent performs preliminary deduplication using weak hash functions before storing chunks in the chunk ID table. This preliminary action reduces the number of unique chunks that need to be tracked, thereby reducing the size of the chunk ID table and minimizing the need for paging operations between RAM and disk.
3Quantity of substance
If deduplication is performed before erasure coding, then storage efficiency is improved, but reliability of unique data decreases
Solution Approach 1:
The patent applies different error protection strategies to different types of data: erasure coding is applied to unique chunks identified during deduplication, while duplicate chunks are protected through reference counting. This local differentiation ensures that storage efficiency is maintained while unique data receives appropriate reliability protection.
Solution Approach 2:
The patent identifies unique chunks during the deduplication phase and applies erasure coding protection beforehand before storing them. This prior cushioning ensures that unique data is protected against future losses while maintaining the storage efficiency benefits of deduplication.
4Reliability
If multiple precoding stages are used to reduce hash collisions, then data integrity is improved, but computational complexity increases
Solution Approach 1:
The patent segments the error protection process into multiple precoding stages applied sequentially: first precoding the input data, then performing deduplication, and finally applying additional precoding to unique chunks. This segmentation allows each stage to be optimized independently, reducing overall computational complexity while maintaining data integrity.
Data Source
AI summary
Methods and apparatus deduplicate and erasure code a message in a data storage system. One example apparatus includes a first chunking circuit that generates a set of data chunks from a message, an outer precoding circuit that generates a set of precoded data chunks and a set of parity symbols from the set of data chunks, a second chunking circuit that generates a set of chunked parity symbols from the set of parity symbols, a deduplication circuit that generates a set of deduplicated data chunks by deduplicating the set of precoded chunks or the set of chunked parity symbols, an unequal error protection (UEP) circuit that generates an encoded message from the set of deduplicated data chunks, and a storage circuit that controls the data storage system to store the set of deduplicated data chunks, the set of parity symbols, or the encoded message.