Pre-warm-up Procedure for Storage Systems to Reduce Data Access Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems experience significant latency when retrieving data from disk drives, especially when metadata is required, leading to varying response times and I/O operation inefficiencies.
Innovation Solution
Implementing a pre-warm-up procedure that forecasts host data access patterns and proactively moves frequently accessed data and metadata from secondary storage to an extension cache, such as flash memory, to reduce latency and stabilize I/O operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If data is loaded from disk drives to main memory on-demand, then storage capacity is maintained, but response time increases significantly (from nanoseconds to milliseconds)
Solution Approach 1:
The system performs preliminary actions by proactively loading frequently accessed data and metadata from disk drives into main memory before they are actually needed. The forecasting mechanism predicts future access patterns and pre-loads corresponding data blocks, transforming on-demand sequential access into proactive parallel operations, thereby reducing response time from milliseconds to nanoseconds.
Solution Approach 2:
The storage system segments data into frequently accessed portions and less frequently accessed portions. By identifying and separating hot data patterns through forecasting, the system loads only the necessary segments into main memory while keeping other data on disk drives, optimizing the balance between response time and storage capacity.
2Productivity
If metadata is retrieved from disk drives to determine data location, then complete data access is enabled, but I/O operations become significantly slower
Solution Approach 1:
The system performs preliminary action by pre-loading metadata (such as location information and access patterns) into main memory during low-I/O periods. This allows the metadata to be instantly available when data access is needed, eliminating the slow sequential retrieval process from disk drives and significantly improving I/O operation speed.
3Loss of time
If data is pre-loaded into main memory, then response time is reduced, but storage capacity for cold data is reduced
Solution Approach 1:
The system segments storage into two distinct layers: a small high-speed layer (main memory) for frequently accessed hot data, and a large low-speed layer (disk drives) for cold data. By using forecasting to identify hot data patterns, the system maintains small footprint in main memory while preserving large capacity on disk drives, optimizing both response time and storage capacity.
Solution Approach 2:
The system dynamically adjusts which data resides in main memory based on forecasted access patterns. Rather than statically allocating fixed storage spaces, the system continuously monitors I/O operations and reconfigures data distribution between memory and disk drives, allowing flexible optimization of both response time and capacity utilization.
4Stability of the object's composition
If pre-warm-up procedure is implemented, then I/O operations are stabilized, but additional processing complexity is introduced
Solution Approach 1:
The system implements self-service by using its own I/O operation statistics to automatically generate forecasts and trigger pre-loading actions. The forecasting mechanism analyzes past I/O patterns and autonomously determines future access needs, eliminating the need for external intervention or complex manual configuration while achieving stable I/O operations.
Data Source
AI summary
A technique operates data storage equipment. The technique involves, during a first time period and while processing circuitry performs host I/O operations on behalf of a set of hosts, performing a data identification operation which provides a data identification result identifying particular data which is routinely accessed by the processing circuitry. The technique further involves, during a second time period after the first time period and in response to the data identification result, copying the particular data from secondary storage to an extension cache which forms part of primary storage to pre-fetch the particular data from the secondary storage for subsequent use by the processing circuitry. The technique further involves, during a third time period after the second time period and while the processing circuitry performs further host I/O operations on behalf of the set of hosts, accessing the particular data from the extension cache.