Dynamic Data Grain Adjustment for Flash Storage Metadata Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In full flash data storage systems, managing massive amounts of data efficiently is challenging due to high concurrent data access and query issues, which are exacerbated by the need for extensive meta data management, leading to increased latency and reduced throughput.
Innovation Solution
A data processing method that dynamically adjusts data granularity during write and read operations by creating new data units and merging data blocks based on logical address occupancy, reducing the amount of meta data generated and optimizing mapping relationships to minimize latency and enhance throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is divided into smaller logical blocks to increase mapping relationships, then data access flexibility is improved, but the amount of meta data generated increases, leading to higher management costs and latency
Solution Approach 1:
The patent merges multiple small logical blocks into larger data units dynamically. When consecutive logical blocks are identified, they are combined into a single data unit with one mapping relationship, reducing the total number of meta data entries while maintaining access flexibility through logical block-level addressing.
Solution Approach 2:
The patent implements dynamic data unit merging where the granularity and size of data units are adjusted based on access patterns and workload characteristics. The system can dynamically merge or split data units to optimize between meta data reduction and access efficiency under different conditions.
2Manufacturing precision
If more meta data is generated to support finer-grained data management, then data organization precision is improved, but the time consumed for meta data management operations increases, affecting throughput
Solution Approach 1:
By merging multiple logical blocks into single data units, the patent reduces the number of mapping relationships that need to be managed. This maintains sufficient data organization precision through logical block addressing while significantly reducing the time required for meta data management operations like insertion, mirroring, and disc flashing.
3Device complexity
If data units are created with fixed granularities, then system complexity is reduced, but the adaptability to different data access patterns decreases, affecting performance
Solution Approach 1:
The patent employs dynamic data unit merging that adapts to different access patterns and workload characteristics. The system can adjust data unit granularity in real-time based on access patterns, optimizing performance for both sequential and random access scenarios while maintaining manageable system complexity through standardized merging operations.
Data Source
AI summary
Provided is a data processing method, which includes: in response to a logical volume receiving a write request, whether a logical address carried in the write request is occupied by a data unit in the logical volume is determined; if not, a data grain which is closest to the size of a data block and is greater than the size of the data block is determined; a new data unit is created in the logical volume by use of the logical address as an initial address and by use of the closest data grain as the length, and a logical address range occupied by the data block in the new data unit is recorded; the data block is written into an underlying storage and a written physical address is returned; and a mapping relationship between the initial address and the physical address is established and saved.