Dynamic In-Memory Ingress Tablet for Storage Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional storage mapping techniques face issues with metadata write amplification and contention, leading to reduced throughput, increased latency, and premature wear on storage media, limiting the performance and reliability of virtual volume systems.
Innovation Solution
The implementation of a mapping apparatus that includes a pre-mapper with a dynamic in-memory ingress tablet and persisted frozen tablets, utilizing a Cuckoo Filter and Broad Operation Database to minimize metadata writes and contention, thereby reducing metadata write amplification and contention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional mapping techniques are used, then storage mapping functionality is provided, but metadata write amplification increases and contention occurs
Solution Approach 1:
The mapping system is segmented into multiple independent mapping tablets (e.g., tablet 0, tablet 1, tablet 2) that can be accessed in parallel. Each tablet stores a portion of the mapping data, allowing write operations to be distributed across multiple tablets simultaneously. This segmentation eliminates the bottleneck of single-metadata-structure updates, reducing metadata write amplification while maintaining high write throughput.
Solution Approach 2:
The system pre-allocates and pre-structures multiple mapping tablets and a mapping index before write operations begin. The mapping index is pre-configured with pointers to all mapping tablets, enabling immediate parallel access during write operations without requiring dynamic structure creation or reorganization. This preliminary setup eliminates runtime overhead and reduces metadata write amplification.
2Reliability
If conventional mapping techniques are used, then storage mapping is achieved, but latency increases due to contention
Solution Approach 1:
By dividing the mapping data into multiple independent tablets, the system eliminates contention that would occur with a single centralized mapping structure. Multiple threads can simultaneously update different tablets without locking conflicts, significantly reducing latency while maintaining mapping accuracy through the coordinated use of the mapping index.
Solution Approach 2:
The mapping index serves as an intermediary structure that coordinates access to multiple mapping tablets. It stores pointers to the tablets and manages the mapping logic, allowing write operations to be routed to appropriate tablets without direct contention between operations. This intermediary layer decouples the coordination overhead from the actual data writing, reducing latency.
3Ease of operation
If conventional mapping techniques are used, then basic storage mapping is provided, but wear on physical storage media increases
Solution Approach 1:
The mapping data is segmented across multiple tablets, allowing the system to distribute write amplification across separate physical storage locations. This segmentation enables more efficient use of storage media by reducing the frequency of writes to any single location, thereby extending media lifetime while maintaining operational simplicity through the unified mapping interface.
Solution Approach 2:
The system implements a mapping tablet lifecycle where tablets can be marked as obsolete and recovered for reuse. When mapping tablets reach capacity or become obsolete, they are discarded and new tablets are created, allowing the system to manage storage media wear by rotating through multiple tablets rather than continuously rewriting to the same locations.
Data Source
AI summary
A mapping apparatus comprises a mapper that translates from an input key to an output key in one or more storage devices. A pre-mapper for processing update operations comprises a plurality of mapping tablets and an in-memory look-up filter to identify a given mapping table storing a given input key. The mapping tablets comprise at least one dynamic in-memory ingress tablet and a plurality of persisted frozen tablets. For a given update operation, a first entry is added to one dynamic in-memory ingress tablet comprising the input key for the given update operation and a corresponding output key where data for the given update operation is stored; and a second entry is added to the look-up filter comprising the input key of the first entry and an identifier of the dynamic in-memory ingress tablet storing the first entry for the given update operation. The dynamic in-memory ingress tablet is persisted as a persisted frozen tablet. Space accounting accounts for usage of the storage devices by a given write operation of a given data offset as one of a new write operation and an overwrite operation.