Erasure-Coded File System Compaction with In-Place Valid Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional erasure-coding-based compaction mechanisms in distributed file systems are inefficient in terms of computing resources and network bandwidth, and create a burden for garbage collection due to the need to recycle and re-write large amounts of invalidated data.
Innovation Solution
The system regroups valid data into new chunks with error-correction-coding protection, maintaining the original physical location of valid data slices and writing new data and parity into the same storage nodes as the original, reducing the amount of data to be written and decoupling compaction from garbage collection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional erasure-coding-based compaction mechanisms are used to merge chunks with holes, then data compaction is achieved, but computing resources and network bandwidth are consumed inefficiently
Solution Approach 1:
The patent extracts only the necessary operations for compaction by identifying and processing only the invalid data portions rather than moving all data. The system selectively reads only invalid data slices from source chunks and writes only the necessary valid data and new parity, leaving valid data in place. This extraction approach minimizes unnecessary data movement and computing operations.
Solution Approach 2:
The compaction process is segmented into distinct operations: identifying invalid data slices, reading only those specific slices, determining target locations, and writing only the necessary data. This segmentation allows the system to process compaction in discrete, optimized steps rather than moving entire chunks of data, reducing overall computing resource consumption.
2Productivity
If conventional erasure-coding-based compaction mechanisms are used to merge chunks with holes, then data compaction is achieved, but network bandwidth is consumed inefficiently
Solution Approach 1:
The patent extracts and processes only the minimal necessary data for compaction by reading only invalid data slices and writing only valid data and new parity. This extraction strategy significantly reduces the volume of data transmitted over the network compared to conventional methods that move entire chunks, thereby reducing network bandwidth consumption.
Solution Approach 2:
The system performs preliminary identification of invalid data slices before initiating data transfer operations. By pre-determining which data slices need to be moved and where they should go, the system avoids unnecessary network traffic and prepares the compaction process in advance, reducing overall bandwidth consumption during the actual compaction execution.
3Productivity
If conventional compaction mechanisms are used, then chunks with holes are merged, but a burden is created for garbage collection in physical drives due to large amounts of invalidated data
Solution Approach 1:
The patent extracts and processes only the minimal necessary data for compaction by identifying and moving only invalid data slices rather than invalidating large portions of data. This selective approach reduces the total volume of data that requires garbage collection, thereby reducing the burden on physical drives while still achieving effective chunk merging.
Solution Approach 2:
The compaction process applies local quality optimization by treating different data slices differently based on their validity status. Valid data slices are left in place without invalidation, while only invalid slices are processed for relocation. This localized approach minimizes the overall impact on garbage collection operations compared to blanket invalidation strategies.
Data Source
AI summary
One embodiment described herein provides a system and method for data compaction in a storage system comprising a plurality of storage nodes. During operation, in response to determining that data compaction is triggered, the system regroups valid data from a first set of data chunks stored in the storage system into a second set of data chunks such that a respective data chunk from the second set of data chunks comprises contiguous valid data slices. The system further performs error-correction-coding protection on the second set of data chunks. A physical location associated with a respective valid data slice remains unchanged subsequent to the error-correction-coding protection.