Count-Min-Sketch Data Tiering for Flash Storage Locality

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing storage array systems face inefficiencies in dynamically managing data tiers based on access patterns, leading to suboptimal performance and resource utilization due to limitations in capturing temporal and spatial locality of data access.

Innovation Solution

Implementing a tiering engine with primary and secondary count-min-sketches to monitor and adjust data placement across hierarchical tiers based on temporal and spatial access patterns, using a translation layer in solid-state drives to generate hints for promoting or demoting data extents.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If traditional storage tiering methods are used to manage data access frequency, then data can be organized into hierarchical tiers, but the system cannot accurately capture temporal and spatial locality of data access patterns

Engineering Contradiction:

Improveaccess pattern capture accuracyVSAvoidtiering system complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent introduces count-min-sketch data structures as intermediary components between the storage hardware and tiering management system. These sketches act as compact mediators that capture access patterns (temporal and spatial locality) without requiring complex full-scale monitoring, thus improving measurement precision while controlling system complexity through probabilistic approximation.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If data extents are frequently promoted and demoted between tiers based on access patterns, then storage performance is optimized, but SSD lifespan is reduced due to increased write operations

Engineering Contradiction:

Improvestorage access performanceVSAvoidSSD lifespan

Core Design Contradiction:

ProductivityVSDuration of action of stationary object

Solution Approach 1:

The patent applies partial action by using count-min-sketches that provide probabilistic rather than exact access pattern measurement. This partial precision allows the system to identify hot data with sufficient accuracy for tiering decisions without the overhead of complete monitoring, thereby reducing unnecessary promote/demote operations that would harm SSD lifespan while still improving storage performance.

Inventive Principle:

Principle #16Partial or excessive action

3Measurement precision

If exact access counting is implemented for every data block, then precise tiering decisions can be made, but memory consumption and processing overhead increase significantly

Engineering Contradiction:

Improveaccess count accuracyVSAvoidmemory consumption

Core Design Contradiction:

Measurement precisionVSQuantity of substance

Solution Approach 1:

The patent employs count-min-sketches which are memory-efficient, probabilistic data structures that use fixed-size arrays with hash functions. These sketches provide approximate but sufficient access counting with minimal memory footprint, effectively replacing expensive exact counting mechanisms while maintaining adequate precision for tiering decisions.

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

Data Source

PatentUS11144224B2Locality-aware, memory-efficient, time-efficient hot data identification using count-min-sketch for flash or streaming applications

Publication Date: 2021.10.12 EMC IP HLDG CO LLC

AI summary

Extents of data are promoted or demoted between hierarchical storage tiers based on temporal locality of access captured with a primary count-min-sketch based on monitored input-output commands. Extents may also be promoted based on spatial locality of access captured with a secondary count-min-sketch based on the monitored input-output commands. The count-min-sketches may be implemented in a tiering engine of a storage array or in the flash translation layers of solid-state drives. The flash translation layers may generate tiering hints that are provided to a tiering engine.