Predictive Data Prefetching Across Storage Tiers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data retrieval from lower performance storage tiers, such as cold storage, results in increased latency and monetary costs due to the need for up-tiering data when accessed, which can disrupt users and is inefficient in managing system load.
Innovation Solution
A data platform uses predictive analysis and machine learning models to prefetch data from secondary storage tiers to primary storage tiers based on predicted access times, reducing latency and managing system load by ensuring data is available when needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If data is stored on lower performance storage tiers to reduce monetary costs, then storage costs are reduced, but data retrieval time increases
Solution Approach 1:
The system performs preliminary actions by prefetching data from secondary storage to primary storage before the data is actually requested by users. The predictive analysis identifies data that will be needed soon and initiates retrieval operations in advance, so that when users request the data, it is already available on high-performance storage, thus maintaining fast retrieval while allowing data to reside on cost-effective storage most of the time
2Loss of time
If data is up-tiered from secondary storage to primary storage when accessed, then data retrieval time is reduced, but system load increases and users experience disruption
Solution Approach 1:
Instead of performing the up-tiering action at the moment of data access, the system performs it preliminarily by predicting which data will be needed and transferring it to primary storage in advance. This spreads the system load over time rather than concentrating it during peak access periods, avoiding user disruption while maintaining fast retrieval performance
Solution Approach 2:
The system uses predictive analysis that processes data access records and user behavior patterns to generate predictions about future data access. This feedback mechanism allows the system to proactively identify and prefetch data that users are likely to need, optimizing the timing of up-tiering operations to balance performance requirements with system load capacity
3Loss of time
If data is stored on high performance primary storage tier, then data retrieval time is reduced, but storage costs increase
Solution Approach 1:
The system keeps data on cost-effective secondary storage and performs preliminary prefetching only for data that predictive analysis identifies as likely to be needed soon. This ensures that primary storage contains only the subset of data that requires fast access, minimizing storage costs while maintaining fast retrieval for active data
Solution Approach 2:
The system applies different storage quality characteristics to different data based on predicted access patterns. Frequently accessed data is kept on high-performance primary storage, while infrequently accessed data remains on lower-cost secondary storage. This local differentiation of storage quality optimizes the balance between retrieval performance and storage cost
Data Source
AI summary
Techniques are disclosed for prefetching data using predictive analysis. An example method comprises storing, by a data platform implemented by a computing system, objects of a file system, wherein a first subset of the objects is stored to a first storage tier and a second subset of the objects is stored to a second storage tier, classifying objects into one or more classifications, storing a data access record for the objects, applying a machine learning model to generate a prediction of future data access to one or more objects of the second subset based on the one or more classifications and the data access record, wherein the prediction includes a predicted time for the future data access, and retrieving, based on the prediction, the one or more objects of the second subset from the second storage tier prior to the predicted time.