Self-Learning Storage Tier Assignment via Metadata Access Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large-scale distributed object storage environments, existing systems face inefficiencies in data storage and retrieval due to inadequate classification and tier assignment of digital objects, leading to reduced performance and increased access times.
Innovation Solution
A self-learning classification manager automatically classifies objects based on metadata fields, tracks access histories, and predicts access frequency and lifetime to proactively assign objects to appropriate storage tiers, such as faster solid-state or slower electromechanical storage, without human input or prior knowledge.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If objects are stored in a multi-tiered storage system without proactive classification and tier assignment, then storage capacity is utilized, but access times increase and system performance deteriorates
Solution Approach 1:
The system performs preliminary classification and tier assignment for new objects based on metadata analysis and predicted access probability distributions before actual access occurs. This proactive approach prevents future access delays by pre-optimizing object placement, resolving the contradiction between fast access and system complexity.
Solution Approach 2:
The classification manager automatically monitors access patterns, updates probability distributions, and reassigns objects without human intervention. The system serves itself by continuously learning from access histories and autonomously optimizing storage tier assignments, reducing the perceived complexity while improving access speed.
2Speed
If objects are frequently migrated between storage tiers to optimize access, then access times improve, but write wear on solid-state devices increases
Solution Approach 1:
The system continuously monitors actual access patterns and compares them with predicted distributions, using this feedback to refine probability models. This allows the system to make more accurate tier assignments initially, reducing the need for frequent migrations and thereby minimizing write wear while maintaining fast access times.
Solution Approach 2:
The system changes the parameter of access probability distribution thresholds dynamically, adjusting when tier migrations occur based on confidence levels in predictions. By setting appropriate thresholds, the system avoids unnecessary migrations that would cause write wear while ensuring objects are placed in optimal tiers for fast access.
3Measurement precision
If detailed access histories are tracked and amalgamated for all objects, then prediction accuracy improves, but CPU and memory resources increase
Solution Approach 1:
The system segments access history tracking by grouping objects with similar metadata characteristics together. Instead of maintaining separate detailed histories for every individual object, it amalgamates access patterns for objects sharing common metadata fields, achieving sufficient prediction accuracy while significantly reducing the computational and memory resources required.
Solution Approach 2:
The system tracks access histories with sufficient detail to achieve accurate predictions but avoids excessive tracking of every possible parameter. It focuses on the most influential metadata fields and access patterns that provide the greatest prediction accuracy improvement per unit of computational resource consumed.
Data Source
AI summary
Multiple objects in a multi-tiered storage system are classified according to multiple metadata fields. Access histories of the classified objects are tracked. Tracked access histories for objects having given values for given metadata fields are amalgamated. Access probability distributions are determined for objects having given values for given metadata fields, based on the amalgamated access histories. Responsive to a new object being added to the system, the new object is classified according to multiple metadata fields. For each specific one of the multiple metadata fields according to which the new object is classified, the determined access probability distribution for objects having the same metadata classification value as the new object is retrieved. The predicted access probability distribution for the new object is determined, by combining the retrieved access probability distributions. The new object is assigned to and stored in a specific storage tier based on the predicted access probability distribution.