Stub Files for Hierarchical Storage Preview
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The indiscriminate access to files by applications like 'Finder' leads to recall storms, causing unnecessary data movement from offline storage to disk storage, which can overload systems, increase disk space usage, and incur additional data transfer costs, wear on storage media, and latency, especially during simple queries.
Innovation Solution
A method involving a machine learning engine that provides a stub profile indicating the offset and length of file regions for previewing, allowing only necessary data to be migrated from the file system to offline storage, thereby reducing recall storms by customizing stub files based on observed I/O patterns and user feedback.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If applications access all files indiscriminately to generate preview images, then preview functionality is improved, but recall storms occur causing system overload and excessive disk space usage
Solution Approach 1:
The patent extracts only the necessary preview data from files and stores it separately in stub files. When applications need preview images, they access only this extracted preview data rather than the entire file, preventing recall storms while maintaining preview functionality.
Solution Approach 2:
The system performs preliminary action by pre-generating and storing preview data in stub files before applications need it. This allows applications to access previews without triggering recalls, as the preview data is already prepared and available in the stub files.
2Productivity
If larger stub regions are provided to mitigate recall storms, then recall storms are reduced, but the effectiveness of migrating data to offline storage is reduced
Solution Approach 1:
The patent applies local quality by making stub files specific to each file type and application needs. Instead of using uniform large stub regions for all files, the system generates customized stub files containing only the preview data relevant to each specific file type (e.g., thumbnail for images, first page for documents), minimizing stub file size while preventing recall storms.
Solution Approach 2:
The system changes parameters by dynamically adjusting stub file content based on file type, application requirements, and observed access patterns. The machine learning engine modifies stub file generation parameters to include only necessary preview portions, optimizing the balance between preventing recall storms and maintaining data migration effectiveness.
3Reliability
If cloud storage recall is triggered for every file access, then data availability is improved, but additional data transfer costs and wear on storage media occur
Solution Approach 1:
The patent extracts preview data from the main file and stores it in stub files on the file system. When applications access files for preview purposes, they read from the local stub files rather than triggering recalls from cloud storage, eliminating unnecessary data transfer costs while maintaining data availability for previews.
Solution Approach 2:
Stub files act as intermediaries between the file system and cloud storage. They provide a local cache of preview data that mediates access requests, allowing applications to obtain preview images without directly triggering cloud storage recalls, thus reducing data transfer costs and storage media wear.
4Productivity
If stub files are created for all files, then recall storms are prevented, but storage space in the file system increases
Solution Approach 1:
The patent applies local quality by creating stub files selectively based on file type and application needs rather than for all files uniformly. The machine learning engine analyzes each file to determine what preview data is necessary, creating stub files only where needed and with minimal necessary content, thus preventing recall storms while minimizing file system storage space usage.
Data Source
AI summary
Intelligent management of stub files in hierarchical storage is provided by: in response to identifying a file to migrate from a file system to offline storage, providing metadata for the file to a machine learning engine; receiving a stub profile for the file from the machine learning engine that indicates an offset from a beginning of the file and a length from the offset for previewing the file; and migrating the portion of the file from the file system to an offline storage based on the stub profile. In some embodiments this further comprises: monitoring file system operations; in response to detecting a read operation of the portion of the file: determining a file type; providing file data to the machine learning engine; and performing a supervised learning operation based on the file type and the file data to update the machine learning engine.