Streaming File Access Prediction for Adaptive Data Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern data storage systems face challenges in accurately predicting future file access patterns due to continuously changing workloads, leading to reduced service performance from offline-trained models that consume significant computing and storage resources.
Innovation Solution
Implementing a streaming model that performs online training using real-time access data features to predict and adapt to changing file access modes, reducing resource consumption while improving prediction accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If an offline-trained model is used to predict file access patterns, then the model can process historical access data, but the prediction accuracy decreases when workloads continuously change and the computing power and storage resources consumed are excessive
Solution Approach 1:
The patent transforms the static offline-trained model into a dynamic online-learning model that continuously updates its parameters in real-time based on incoming access data. This allows the model to adapt to changing workloads dynamically, maintaining high prediction accuracy without requiring excessive computing resources for retraining from scratch.
Solution Approach 2:
The patent implements continuous learning where the model processes access data streams continuously rather than undergoing periodic batch retraining. This continuous update mechanism ensures the model always reflects current access patterns while avoiding the resource-intensive interruptions of complete model retraining.
2Productivity
If an offline-trained model is used to predict next access parts and access popularity, then historical access data can be processed, but the service performance of the data storage system decreases due to low accuracy in continuously changing file access modes
Solution Approach 1:
The patent incorporates feedback mechanisms where the model uses actual access outcomes to adjust and refine its predictions. By comparing predicted access patterns with actual access behavior, the model continuously improves its accuracy, thereby enhancing service performance in response to changing workloads.
Solution Approach 2:
The model transitions from a static prediction approach to a dynamic adaptive approach that responds to real-time changes in access patterns. This dynamic behavior enables the system to maintain high service performance despite continuously changing file access modes.
3Measurement precision
If a streaming model performs online training using real-time access data features, then the prediction accuracy improves and resource consumption is reduced, but the system complexity increases
Solution Approach 1:
The patent extracts and processes only the essential features from access data streams, filtering out redundant information. This feature extraction approach simplifies the online training process and reduces computational complexity while maintaining prediction accuracy.
Solution Approach 2:
The patent segments the complex prediction task into manageable components, processing different aspects of access data separately and combining results. This segmentation strategy reduces the complexity of handling complex access patterns in real-time.
Data Source
AI summary
A data storage system trains a first streaming model based on a plurality of first access data features corresponding to a first file in an (i−1)th access request and a first access attribute parameter of the first file; then inputs a plurality of second access data features corresponding to the first file in an ith access request into the first streaming model, to obtain a second access attribute parameter of the first file in an (i+1)th access request; and then pre-fetches or migrates the first file based on the second access attribute parameter.