Storage Apparatus Data Routing by Write Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage methods face challenges in reducing write latency for Write Ahead Logs (WAL), log data, metadata, and upper-layer application data, which affects overall data throughput.
Innovation Solution
A data storage method that utilizes a storage apparatus with multiple storage units, where data is written to a high-performance storage unit for low-latency requirements and a lower-performance storage unit for higher-latency requirements, based on stream ID conditions that indicate write latency requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If a single storage unit is used for all data types, then device complexity is reduced, but write latency cannot be optimized for different data types
Solution Approach 1:
The storage apparatus is divided into multiple storage units (first storage unit and second storage unit) with different performance characteristics. The first storage unit has higher read/write performance while the second storage unit has lower performance, allowing different data types to be stored in appropriate units based on their latency requirements
Solution Approach 2:
Different storage units are assigned different performance characteristics tailored to specific data types. The first storage unit is optimized for high-performance data (WAL, log data, metadata) while the second storage unit handles lower-performance data, creating local optimization rather than uniform performance throughout the system
2Loss of time
If all data is written to high-performance storage unit, then write latency is reduced, but storage capacity and cost efficiency are compromised
Solution Approach 1:
Data is segmented into different streams based on their write latency requirements. Critical data requiring low latency (WAL, log data, metadata) is directed to the first storage unit, while non-critical data can use the second storage unit, optimizing both performance and capacity utilization
Solution Approach 2:
The system changes the storage parameter (performance level) based on data type characteristics. By identifying data streams with different latency requirements and assigning them to appropriate storage units, the system achieves optimal performance for critical data while maintaining cost-effectiveness for less critical data
3Productivity
If stream ID based routing is implemented, then data throughput is improved, but control complexity increases
Solution Approach 1:
A stream ID is assigned to each data stream to encode its write latency requirements. The storage apparatus uses this parameter (stream ID) to automatically route data to the appropriate storage unit, simplifying the control logic compared to complex per-data-type routing rules
Solution Approach 2:
The system uses the stream ID as a feedback mechanism that carries information about data characteristics directly in the data stream itself, allowing the storage apparatus to make intelligent routing decisions without complex external control signals
Data Source
AI summary
A data storage method includes: in response to a stream ID carried by an IO write request of a host satisfying a first preset condition, writing data corresponding to the IO write request into a first storage unit; and in response to the stream ID carried by the IO write request satisfying a second preset condition, writing the data corresponding to the IO write request into a second storage unit, wherein the stream ID indicates write latency requirement information of the data corresponding to the IO write request, wherein a data write latency indicated by the stream ID satisfying the first preset condition is less than the data write latency indicated by the stream ID satisfying the second preset condition, wherein a read and write performance of the first storage unit is higher than the read and write performance of the second storage unit.