Log-structured File System Segment Cleaning via Data Block Remapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional erasure coding in distributed data systems, such as RAID 6 schemes, face inefficiencies in writing data due to high read and write amplification factors, leading to increased I/O latency and performance limitations.
Innovation Solution
A distributed-computing system with a capacity storage tier using erasure-coded data stripes and a performance storage tier for metadata, employing a B-tree structure and logical address mapping to optimize data storage and retrieval, allowing for reduced read and write operations through full or partial stripe write modes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional erasure coding (RAID 6) is used for fault-tolerant distributed storage, then data reliability is improved, but I/O latency and write amplification increase
Solution Approach 1:
The system segments storage into two distinct tiers: a performance storage tier using RAID 1 mirroring for low-latency operations and a capacity storage tier using RAID 6 erasure coding for high-capacity fault-tolerant storage. This segmentation allows each tier to be optimized for its specific function, resolving the contradiction between reliability and I/O latency by handling different data types in different locations.
Solution Approach 2:
A logical-to-physical address mapping layer acts as an intermediary between the host and the two storage tiers. This mapping layer intelligently directs I/O operations to the appropriate tier based on data characteristics and access patterns, enabling the system to achieve both low latency (by routing hot data to performance tier) and high reliability (by storing data with EC protection in capacity tier).
2Reliability
If RAID 6 erasure coding is used for distributed storage, then fault tolerance is improved, but write amplification factor increases
Solution Approach 1:
The write operation is segmented into two paths: performance-critical writes go to the RAID 1 performance tier with minimal amplification, while capacity-optimized writes go to the RAID 6 capacity tier. This segmentation reduces overall write amplification by avoiding EC recalculations for all writes, only performing them when data is migrated to or refreshed in the capacity tier.
Solution Approach 2:
The system performs preliminary actions by maintaining a logical-to-physical address mapping that tracks data locations and validity. This mapping allows the system to determine whether writes need to trigger EC recalculations in advance, avoiding unnecessary write operations and reducing write amplification factor by only performing EC operations when absolutely necessary.
3Quantity of substance
If data is striped and erasure coded across multiple disks, then storage capacity efficiency is improved, but read and write operation complexity increases
Solution Approach 1:
The logical-to-physical address mapping serves as an intermediary that abstracts the complexity of erasure-coded striping from host operations. The mapping layer handles all the complex logic for determining which disks contain valid data, reconstructing lost blocks, and managing stripe operations, thereby reducing operation complexity at the host level while maintaining high storage capacity efficiency through EC.
Solution Approach 2:
The capacity storage tier employs self-service mechanisms where the RAID 6 controller automatically performs erasure coding calculations, data reconstruction, and validity checking without requiring complex host-level intervention. This self-service approach simplifies host operations while maintaining efficient use of storage capacity through automated EC management.
Data Source
AI summary
Client data is structured as a set of data blocks. A first subset of data blocks is stored on a current segment of the disks. A second subset of data blocks is stored on a previous segment. A request to clean client data is received, including a request to update the current segment to include the second subset of data blocks. The second subset of data blocks is accessed and transmitted from a lower layer to a higher system layer. Parity data is generated at the higher layer. The parity data is transmitted to the lower layer. The lower layer updates second mapping data. In the updated mapping of the second mapping data, each local address that references a data block of the second subset of data blocks is included in the current segment of the plurality of disks. The lower layer writes the parity data in the current segment.