Key-Value Store Selective Journaling Reduces Write Amplification

Solution Overview

Problem

Conventional key-value store (KVS) systems using journaling for data consistency and durability often result in increased write operations and inefficient bandwidth utilization due to the need to write data multiple times to persistent storage, leading to sub-optimal input/output operations.

Innovation Solution

Implementing a KVS with a selective journaling approach that uses adaptive data structures and metadata logs to store key-value pairs, allowing for reduced write operations by avoiding redundant data writes and optimizing storage formats based on criteria such as data size and observed I/O patterns.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If conventional journaling is used to ensure data consistency and durability, then data reliability is improved, but write operations increase and bandwidth utilization deteriorates

Engineering Contradiction:

Improvedata consistency and durabilityVSAvoidwrite operations and bandwidth utilization

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent applies local quality by differentiating the treatment of key data and value data. Key data is always written to the journal for consistency, while value data is selectively written based on size criteria. This localized differentiation resolves the contradiction by applying journaling only where necessary for reliability (keys) and avoiding it where it causes inefficiency (large values).

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The patent changes the parameter of data storage location based on data characteristics. Small values are stored in the journal (metadata log) while large values are stored separately. This parameter change resolves the contradiction by adapting the storage strategy to the specific data parameters, reducing unnecessary writes of large values while maintaining consistency for keys and small values.

Inventive Principle:

Principle #35Parameter changes

2Reliability

If data is written multiple times to persistent storage for journaling, then data durability is improved, but I/O operations increase and efficiency deteriorates

Engineering Contradiction:

Improvedata durabilityVSAvoidI/O operations time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent applies local quality by differentiating the treatment of key data and value data. Key data is always written to the journal for consistency, while value data is selectively written based on size criteria. This localized differentiation resolves the contradiction by applying journaling only where necessary for reliability (keys) and avoiding it where it causes inefficiency (large values).

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The patent applies partial action by writing only a subset of data (keys and small values) to the journal, rather than all data. This partial approach resolves the contradiction by achieving sufficient durability through selective journaling while avoiding the excessive I/O overhead of writing all data multiple times.

Inventive Principle:

Principle #16Partial or excessive action

3Productivity

If selective journaling with adaptive data structures is used, then write amplification is reduced, but system complexity increases

Engineering Contradiction:

Improvewrite amplificationVSAvoidadaptive data structures

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent changes the parameter of data storage location based on data characteristics. Small values are stored in the journal (metadata log) while large values are stored separately. This parameter change resolves the contradiction by adapting the storage strategy to the specific data parameters, reducing unnecessary writes of large values while maintaining consistency for keys and small values.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent segments data storage into two distinct locations: the journal (metadata log) for keys and small values, and separate storage for large values. This segmentation resolves the contradiction by allowing optimized handling of different data types, reducing write amplification for large values while maintaining consistency for keys.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS11334270B2Key-value store using journaling with selective data storage format

Publication Date: 2022.05.17 MICRON TECHNOLOGY INC

AI summary

Aspects of the present disclosure provide for operations of a key-value store (KVS) that uses journaling with storage of data (e.g., key data or value data of a key-value pair) in different types of data storage objects (e.g., data storage formats) selected based on one or more criteria.