Erasure Coding Garbage Collection for Cloud Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing cloud storage systems incur unnecessary resource consumption due to redundant data replication during the garbage collection process, which results in wasteful use of storage and network resources.
Innovation Solution
Implementing a stream layer that directly converts identified good data into erasure-coded fragments, eliminating the need for data replication by distributing these fragments across multiple server nodes for durability, thereby optimizing disk space and network resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is replicated across multiple nodes during garbage collection, then data durability is improved, but storage and network resource consumption increases
Solution Approach 1:
The patent extracts only the necessary data fragments for durability protection through erasure coding, separating the essential durability function from the redundant replication process. By taking out only the minimal required data (k data fragments plus m parity fragments) and distributing them across nodes, the system achieves durability without unnecessary resource consumption.
Solution Approach 2:
The patent changes the replication parameter from traditional full-data replication to erasure coding with specific fragment counts. Instead of replicating entire extents multiple times, the system uses k data fragments and m parity fragments where k+m > n (number of nodes), achieving durability with reduced storage parameters and optimized resource usage.
2Reliability
If data is replicated across multiple nodes during garbage collection, then data durability is improved, but network resource consumption increases
Solution Approach 1:
The patent extracts only the essential data fragments needed for durability through erasure coding, eliminating unnecessary network traffic associated with replicating complete extents. By transferring only k data fragments and m parity fragments across the network rather than full extents, network resource consumption is significantly reduced while maintaining durability.
Solution Approach 2:
The patent applies partial action by transferring only the necessary fragments (k data + m parity) rather than complete extents. This partial transfer approach provides sufficient durability protection without the excessive network bandwidth consumption that would result from replicating entire extents across multiple nodes.
3Reliability
If traditional replication process is used during garbage collection, then data durability is ensured, but processing time increases
Solution Approach 1:
The patent segments the data into k data fragments and m parity fragments through erasure coding, enabling parallel processing and independent distribution across nodes. This segmentation allows the garbage collection process to operate more efficiently by distributing work rather than replicating entire extents sequentially, reducing overall processing time while ensuring durability.
Solution Approach 2:
The patent performs erasure coding and fragment distribution as preliminary actions during garbage collection, preparing data for durable storage before the extent is fully processed. This preliminary encoding and distribution reduces subsequent processing requirements and accelerates the overall garbage collection timeline while maintaining durability guarantees.
Data Source
AI summary
Provided is a system and method for converting active data identified by a garbage collection operation into erasure coded fragments. In one example, the method may include identifying data blocks in use and interspersed among garbage data blocks not in use in cloud storage based on a garbage collection operation, extracting object data from the identified data blocks in use into a data container while leaving object data of the garbage data blocks not in use, and fragmenting a predetermined amount of extracted object data stored within the data container, the fragmenting comprising converting the predetermined amount of object data into a plurality of fragments including data fragments storing portions of the data and parity fragments for reconstructing the data, and writing the plurality of fragments in a distributed manner among a plurality of storage nodes.