Hardware Pipeline Metadata Caching for Storage Write Amplification

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

NAND flash memory devices, especially those with multiple level cell (MLC) technology, face instability and performance degradation due to severe write amplification when integrated into storage systems with Ceph control architectures, leading to increased I/O latency and reduced lifespan.

Innovation Solution

Implementing a pipeline-based accessing management method within a storage server that utilizes a host device to process user data through a pipeline architecture, caching metadata with a hardware pipeline module outside the storage device layer to prevent write amplification and optimize performance without introducing significant side effects.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If Ceph control architecture is used to manage storage operations, then storage system functionality and data management capability are improved, but write amplification increases significantly causing performance degradation and reduced device lifespan

Engineering Contradiction:

Improvestorage system functionalityVSAvoidI/O performance

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The patent introduces a pipeline architecture as an intermediary layer between the Ceph control architecture and the storage device layer. This pipeline includes multiple processing stages (journaling stage, metadata processing stage, data processing stage) that handle write operations in a coordinated manner, reducing the write amplification effect while maintaining Ceph's data management capabilities

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent segments the write operation processing into distinct pipeline stages: journaling stage for logging operations, metadata processing stage for handling metadata updates, and data processing stage for actual data writes. This segmentation allows each stage to be optimized independently and reduces unnecessary write amplification by processing only essential operations at each stage

Inventive Principle:

Principle #1Segmentation

2Reliability

If metadata operations are performed frequently to maintain data integrity, then data management reliability is improved, but write amplification increases causing increased I/O latency

Engineering Contradiction:

Improvedata management reliabilityVSAvoidI/O latency

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent implements preliminary action by pre-processing metadata operations in the metadata processing stage before actual data writes occur. This includes pre-validating metadata, pre-updating journal entries, and pre-coordinating with the journaling stage, which reduces the need for repeated metadata operations during data writes and thereby reduces I/O latency while maintaining data integrity

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent ensures continuity of useful action by overlapping metadata processing with data processing through the pipeline architecture. While data is being processed in one stage, metadata operations continue in parallel in other stages, eliminating idle waiting time and reducing overall I/O latency without compromising data management reliability

Inventive Principle:

Principle #20Continuity of useful action

3Adaptability or versatility

If fixed block size is used for storage operations, then storage device compatibility is improved, but write amplification occurs when updating small metadata causing performance reduction

Engineering Contradiction:

Improvestorage device compatibilityVSAvoidstorage system performance

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The patent applies local quality by treating metadata operations and data operations differently within the pipeline. Metadata operations are handled with fine-grained control in the metadata processing stage, allowing small metadata updates to be processed efficiently without being forced into fixed block size writes. Data operations maintain fixed block size for compatibility, while journal operations use variable sizing. This localized differentiation eliminates unnecessary write amplification for metadata while preserving storage device compatibility

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS11416403B2Method and apparatus for performing pipeline-based accessing management in storage server with aid of caching metadata with hardware pipeline module during processing object write command

Publication Date: 2022.08.16 SILICON MOTION TECH HONG KONG LTD

AI summary

A method for performing pipeline-based accessing management in a storage server and associated apparatus are provided. The method includes: in response to a request of writing user data into the storage server, utilizing a host device within the storage server to write the user data into a storage device layer of the storage server and start processing an object write command corresponding to the request of writing the user data with a pipeline architecture of the storage server; utilizing the host device to input metadata corresponding to the user data into at least one pipeline within the pipeline architecture; and utilizing the host device to cache the metadata with a first cache module of the pipeline, for controlling the storage server completing the request without generating write amplification of the metadata, wherein the first cache module is a hardware pipeline module outside the storage device layer.