Segment Cleaning by Local Block Copying in Log-Structured Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional erasure coding in distributed data systems, such as RAID 6 schemes, face inefficiencies in writing data due to high read and write amplification factors, leading to increased I/O latency and traffic, which limits storage system performance.
Innovation Solution
A distributed-computing system architecture that includes a capacity tier for storing erasure-coded data stripes across multiple disks and a performance tier for metadata, utilizing a B-tree structure to manage address mapping and log data, allowing for efficient data storage and retrieval by caching client data in the performance tier and updating metadata in the capacity tier, thereby reducing the need for excessive read and write operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional erasure coding (RAID 6) is used for fault-tolerant distributed storage, then data reliability is improved, but I/O efficiency deteriorates due to high read and write amplification factors
Solution Approach 1:
The system segments data into data blocks and organizes them into stripes across multiple disks. Each stripe contains multiple data blocks and corresponding parity blocks, allowing independent processing of individual blocks while maintaining overall fault tolerance. This segmentation enables parallel I/O operations and reduces the amplification factor by limiting the scope of read-modify-write operations to individual blocks rather than entire stripes.
Solution Approach 2:
The patent introduces a log-structured file system as an intermediary layer between the application and the underlying erasure-coded storage. This layer maintains a log of data blocks and their locations, allowing the system to track and manage data blocks independently without requiring full stripe reconstruction for every operation. The log structure acts as a mediator that reduces the complexity of read-modify-write operations in erasure-coded systems.
2Productivity
If data is striped across multiple disks for parallel I/O operations, then I/O efficiency is improved, but fault tolerance deteriorates without redundancy
Solution Approach 1:
The system applies different quality levels to different parts of the storage system. Data blocks are striped across multiple disks for high-speed access, while parity blocks are distributed across the same disks with erasure coding providing localized redundancy. This allows the system to optimize for both speed (through striping) and reliability (through localized parity protection) simultaneously, with each disk serving multiple roles in the overall storage architecture.
Data Source
AI summary
Client data is structured as a set of data blocks. A first subset of data blocks is stored on a current segment of a plurality of disks. A second subset of data blocks is stored on a previous segment. A request to clean client data is received. The request includes a request to update the current segment to include the second subset of data blocks. The second subset of data blocks is accessed and transmitted from a lower layer to a higher layer of the system. Parity data is generated at the higher layer. The parity data is transmitted to the lower layer. The lower layer is employed to generate a local copy of the second subset of data blocks. Each local address that references the local copy of the second subset of data blocks is included in the current segment. The parity data is written in the current segment.