File System Recovery Log Layer for Distributed Storage Initialization

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Storage systems, especially those based on spinning disks, face performance issues and high costs when handling small and synchronous write operations due to the need for expensive non-volatile read and write memory (NVRAM) in every node, and using SSDs can degrade their lifetime if not managed properly.

Innovation Solution

A file system recovery log layer is integrated to buffer data in a cyclical manner, transforming random requests into synchronous sequential updates, with a dynamic threshold determining when data is written to permanent storage or the recovery log, optimizing performance and reducing SSD degradation.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If NVRAM is installed in every node to buffer small writes, then small write performance is improved, but system cost increases significantly

Engineering Contradiction:

Improvesmall write performanceVSAvoidsystem cost

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent introduces a recovery log as an intermediary component between the file system and permanent storage. This recovery log buffers small write operations and transforms them into larger sequential writes, eliminating the need for NVRAM in every node while maintaining performance benefits

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent merges the buffering function with the recovery log mechanism, combining multiple small write operations into larger sequential writes. This consolidation allows the system to achieve NVRAM-like performance without requiring expensive NVRAM hardware in each node

Inventive Principle:

Principle #5Merging (Combining)

2Productivity

If SSDs are used to buffer random writes, then small write performance is improved, but SSD lifetime is degraded

Engineering Contradiction:

Improvesmall write performanceVSAvoidSSD lifetime

Core Design Contradiction:

ProductivityVSDuration of action of stationary object

Solution Approach 1:

The recovery log acts as a protective intermediary that absorbs random write operations and transforms them into sequential writes. This prevents direct random writes to SSDs, thereby extending SSD lifetime while maintaining performance

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent converts the harmful effect of random writes (which degrade SSDs) into a beneficial process by using the recovery log to batch and sequentialize these writes. The random writes are transformed into larger sequential operations that are safe for SSD storage

Inventive Principle:

Principle #22Blessing in disguise (Convert harm into benefit)

3Ease of manufacture

If a fixed threshold is used for determining write size, then implementation is simple, but adaptability to varying workloads is reduced

Engineering Contradiction:

Improveimplementation simplicityVSAvoidworkload adaptability

Core Design Contradiction:

Ease of manufactureVSAdaptability or versatility

Solution Approach 1:

The patent implements a dynamic threshold mechanism that automatically adjusts the write size threshold based on current system conditions and workload characteristics. This allows the system to adapt to varying workloads while maintaining a relatively simple implementation through automated adjustment

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS11429567B2Quick initialization of data regions in a distributed storage system

Publication Date: 2022.08.30 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

A method includes integrating a file system recovery log layer in a file system. The file system buffers data in a cyclical manner, and transforms all incoming random requests into a series of synchronous sequential updates. The method determines a length value of a write transaction byte-range for a received write transaction. Upon the length value exceeding a threshold value, data for the write transaction byte-range is written to permanent storage in the file system. Upon the length value being less than the threshold value, the data for the write transaction byte-range is written to a recovery log that is stored in the file system recovery log layer. The threshold value is changed dynamically while the file system is mounted.