Tiered Storage Data Clustering via Machine Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing tiered data storage systems face bottlenecks in transferring data between high-performance file systems and archival storage, such as object storage, due to inefficiencies in tiering, archiving, and retrieval processes, leading to lag, unnecessary data transfer, and wasteful processing.
Innovation Solution
A data storage system utilizing machine learning logic to determine clusters of associated files, compressing and archiving them, and storing metadata in a key-value store for near-real-time search and restoration, optimizing file transfers and access across tiers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is transferred from high-performance file system to archival storage, then storage cost is reduced, but access speed decreases
Solution Approach 1:
The storage system is segmented into multiple tiers (high-performance file system for active data, archival storage for historical data) with automatic classification. Files are segmented based on access patterns and business value, allowing optimal performance for critical data while cost-effectively storing less critical data in archival tiers.
Solution Approach 2:
The system dynamically adjusts data placement between tiers based on real-time access patterns, business value assessment, and storage costs. Files can be promoted to or demoted from high-performance storage based on changing access requirements, optimizing the balance between performance and cost.
2Reliability
If all files are restored to high-performance file system, then data availability is improved, but processing efficiency decreases
Solution Approach 1:
Different quality levels of service are provided to different files based on their business value and access patterns. High-value files with frequent access remain in high-performance storage, while low-value files are archived. This local differentiation optimizes both availability for critical data and processing efficiency overall.
Solution Approach 2:
The system continuously monitors file access patterns, business value, and storage performance, using this feedback to dynamically adjust data placement decisions. This feedback loop ensures optimal balance between data availability and processing efficiency based on actual system behavior.
3Adaptability or versatility
If custom tiering solutions are implemented, then data management flexibility is improved, but system complexity increases
Solution Approach 1:
The system uses a universal classification framework that handles multiple functions: data tiering, archiving, retrieval, and cost optimization. This multi-functional approach provides flexible data management capabilities while avoiding the complexity of separate custom solutions for each function.
4Productivity
If data is transferred between storage tiers, then storage optimization is improved, but transfer time increases
Solution Approach 1:
The system performs preliminary classification and assessment of files before transfer, grouping them into logical sets based on access patterns and business value. This preliminary organization enables more efficient batch transfers and reduces overall transfer time compared to individual file processing.
Data Source
AI summary
Example tiered storage systems, storage devices, and methods provide intelligent data access across tiered storage systems. An example system can comprise one or more computing devices, a file system, an object storage system comprising an object storage, and a data tiering application. The data tiering application is executable by one or more computing devices to perform operations comprising determining, using machine learning logic, a cluster of associated files stored in the file system; and archiving the cluster of associated files from the file system to the object storage coupled for electronic communication to the file system via a computer network.