Erasure Coding of Replicated Blocks Without Background Scanning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing erasure coding techniques in storage systems are inefficient due to delayed processing, which leads to prolonged storage of replicas before conversion to error correction blocks, resulting in increased storage usage and network traffic.
Innovation Solution
Implementing a method where a pre-designated node receives replicas and, upon a trigger condition, replaces them with error correction blocks, reducing the need for background scanning and minimizing network traffic by performing erasure coding locally on a single node.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If replicas are stored on nodes for a significant amount of time before erasure coding is performed, then data reliability is maintained, but storage efficiency deteriorates and storage space is wasted
Solution Approach 1:
The system pre-designates specific nodes to receive and store replicas before they are needed for erasure coding. This preliminary assignment allows replicas to be quickly identified and processed for erasure coding once written, rather than waiting for background scanning to discover them. The pre-designated nodes maintain replicas in a readily available state for immediate erasure coding conversion, resolving the contradiction by enabling fast transition from replica to error correction block while maintaining reliability.
2Productivity
If background scanning techniques are used to identify candidates for erasure coding, then erasure coding can be performed, but processing time increases and system complexity deteriorates
Solution Approach 1:
Nodes pre-designate specific targets for storing replicas before they are written. When a replica is written, the system immediately knows which pre-designated node should receive it and can initiate erasure coding processes without waiting for background scanning to identify candidates. This eliminates the time-consuming scanning phase and allows erasure coding to begin immediately once the replica is received, significantly improving productivity while reducing processing time.
Solution Approach 2:
The system extracts the background scanning function entirely from the erasure coding process. By pre-designating nodes and using explicit replica routing information, the patent removes the need for periodic background scanning to identify erasure coding candidates. This extraction of the scanning step eliminates the associated time loss and system complexity while maintaining the ability to perform erasure coding efficiently.
3Quantity of substance
If replicas are converted to error correction blocks quickly, then storage efficiency is improved, but network traffic increases due to complex scanning and coordination
Solution Approach 1:
The system pre-establishes which nodes will receive replicas and store them locally. When a replica is written, the writing node already knows the destination and can transfer the replica directly to the pre-designated node without complex coordination or scanning. This preliminary arrangement minimizes network traffic by eliminating the need for discovery protocols and complex coordination mechanisms, while enabling quick conversion to error correction blocks that improve storage efficiency.
4Productivity
If complex scanning techniques are used to identify erasure coding candidates, then comprehensive candidate identification is achieved, but device complexity increases
Solution Approach 1:
The patent removes the complex background scanning mechanism entirely from the system. Instead, nodes maintain pre-designated lists of targets for replica storage, and replicas are routed directly to these predetermined locations. This extraction of the scanning function dramatically reduces system complexity while maintaining accurate candidate identification, as the pre-designated nodes are explicitly known before replicas are written.
Solution Approach 2:
Each node independently maintains its own pre-designated list of targets for receiving replicas. This self-service approach eliminates the need for centralized coordination or complex inter-node communication to identify erasure coding candidates. Nodes autonomously manage their replica storage assignments, reducing overall system complexity while ensuring accurate candidate identification for erasure coding.
Data Source
AI summary
Various embodiments set forth techniques for erasure coding of replicated data blocks. The techniques include receiving, by a pre-designated node, data associated with an erasure coded strip from a first node; receiving, by the pre-designated node, a replica for a first data block; saving the replica in an erasure coded strip; and in response to a trigger condition, replacing, by the pre-designated node, the replica and at least one replica of a second data block with an error correction block.