File System Block Mapping Bypass for Low-Latency Erasure-Coded Writes

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Conventional erasure coding in distributed data systems, such as RAID 6 schemes, face inefficiencies in writing data due to high read and write amplification factors, leading to increased I/O latency and performance limitations.

Innovation Solution

A distributed-computing system with a capacity tier using erasure-coded data stripes and a performance tier for metadata, employing a B-tree address map and partial stripe write modes to reduce the number of read and write operations, and utilizing a log-structured file system to manage data efficiently across multiple disks.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If conventional erasure coding (RAID 6) is used for fault-tolerant distributed storage, then data reliability is improved, but I/O latency and write amplification increase

Engineering Contradiction:
Improvedata reliabilityVSAvoidI/O latency
Core Design Contradiction:
ReliabilityVSLoss of time

Solution Approach 1:

The patent segments the address mapping function into two distinct layers: a first layer mapping (L1) that maps logical addresses to intermediate addresses, and a second layer mapping (L2) that maps intermediate addresses to physical addresses. This segmentation allows the system to bypass the L2 mapping for sequential writes by directly using L1, thereby reducing I/O latency while maintaining data reliability through the preserved L2 mapping structure for fault tolerance.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The patent performs preliminary action by pre-allocating intermediate addresses through the first layer mapping before actual data writing occurs. The L1 mapping is established in advance and can be directly used for sequential writes without requiring real-time L2 lookup, thus reducing I/O latency while the L2 mapping remains available for reliability operations.

Inventive Principle:
Principle #10Preliminary action

2Reliability

If conventional erasure coding with full stripe writes is used, then fault tolerance is improved, but write amplification factor increases

Engineering Contradiction:
Improvefault toleranceVSAvoidwrite amplification factor
Core Design Contradiction:
ReliabilityVSProductivity

Solution Approach 1:

The patent applies partial action by allowing sequential writes to proceed using only the first layer mapping without requiring complete stripe formation or full second layer mapping updates. This partial approach reduces the write amplification factor from 3x (read-modify-write) to接近1x for sequential writes, while fault tolerance is maintained through the underlying erasure coding structure and the preserved L2 mapping for recovery operations.

Inventive Principle:
Principle #16Partial or excessive action

Solution Approach 2:

The patent extracts the address mapping overhead from the write path by separating L1 and L2 mappings. The L2 mapping is extracted out of the critical write path, allowing writes to use only L1 mapping, thus reducing write amplification. The L2 mapping remains intact for fault tolerance operations, achieving both goals simultaneously.

Inventive Principle:
Principle #2Taking out (Extraction)

Data Source

PatentUS11263146B2Efficient accessing methods for bypassing second layer mapping of data blocks in file systems of distributed data systems
Publication Date: 2022.03.01 VMWARE INC
  • US11263146B2 patent drawing
  • US11263146B2 patent drawing
  • US11263146B2 patent drawing

AI summary

Systems for accessing client data is described. A request to access a first data block is received. The request indicates a first logical address referencing the first data block. First mapping data is employed to identify a first physical addresses corresponding to the first logical addresses. The first mapping data encodes a first LOM transaction ID and candidate local addresses. The first mapping data is employed to identify the candidate local address and the first LOM transaction ID. A usage table is employed to determine the current status of the first LOM transaction ID. The candidate local address is employed to access the first data block. Second mapping data is employed to identify an updated local address of the set of local addresses. The updated local address currently references the first data block. The updated local address is employed to access the first data block.