Machine Learning File Archiving with Predictive Cache
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data archiving methods face significant delays and inefficiencies due to network latency and the time required to search and extract archived files, particularly in large corporations with extensive data and user bases.
Innovation Solution
A system and method utilizing machine learning to predict file access patterns, archiving files to a tiered storage system, and retrieving them into a file cache on the storage server based on these predictions, allowing for quicker access and reducing delays.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If files are archived to a tiered storage system, then storage capacity is improved, but access time deteriorates
Solution Approach 1:
The system performs preliminary actions by predicting which archived files will be needed and pre-fetching them into the file cache before actual access requests occur. This anticipatory mechanism reduces access time by having files ready in advance, resolving the contradiction between maintaining large archived storage and enabling fast access.
Solution Approach 2:
The patent introduces a file cache as an intermediary layer between the tiered storage system and users. This intermediary holds copies of predicted archived files, allowing users to access files from the fast cache rather than directly from the slow tiered storage, thus resolving the access time penalty while preserving the storage capacity benefits.
2Productivity
If archived files are kept in tiered storage system, then storage efficiency is improved, but retrieval speed deteriorates
Solution Approach 1:
The system performs preliminary retrieval actions by predicting which files will be needed and fetching them into the cache in advance. This pre-positioning of files maintains storage efficiency while dramatically improving retrieval speed when files are actually needed.
Solution Approach 2:
The patent implements a dynamic system where the file cache contents are continuously updated based on machine learning predictions of access patterns. The cache dynamically adjusts which files to hold, optimizing the balance between storage efficiency and retrieval speed based on changing access requirements.
3Loss of time
If machine learning module predicts and pre-fetches files, then access time is improved, but system complexity deteriorates
Solution Approach 1:
The system employs a self-service mechanism where the machine learning module automatically analyzes access patterns, generates predictions, and triggers file pre-fetching without manual intervention. This automation manages the system complexity internally while delivering the benefit of reduced access time to users.
Solution Approach 2:
The patent implements feedback loops where the machine learning module continuously learns from actual file access patterns and refines its predictions. This feedback mechanism allows the system to adapt and optimize its complexity management, improving access time predictions while the system learns from experience.
Data Source
AI summary
Methods for file archiving using machine learning are disclosed herein. An exemplary method comprises archiving a first file of a plurality of files from a storage server to a tiered storage system, training a machine learning module based on file access operations for the plurality of files, determining one or more rules for predicting access to the archived files using the machine learning module, determining a prediction of access of the archived file based on the one or more rules and retrieving the archived file from the tiered storage system into a file cache in the storage server based on the prediction of access.