Hierarchical Storage Data Migration for AI Workloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In hierarchical storage systems, the server fails to perform appropriate hierarchical control for access from clients executing AI-related workflows like machine learning or data mining, leading to degraded response performance due to differing access tendencies and inefficient use of high-performance storage devices.
Innovation Solution
A control apparatus that includes a processor configured to receive reading requests, measure processing times, and migrate data pieces across different storage tiers based on their reading capabilities and processing times to optimize storage arrangement, ensuring efficient use of high-performance devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is arranged in high-performance storage devices, then response performance is improved, but storage cost increases
Solution Approach 1:
The patent applies local quality by placing frequently accessed data (hot data) in high-performance storage devices while storing less frequently accessed data (cold data) in low-performance storage devices. The storage system dynamically identifies access patterns and assigns data to appropriate storage tiers based on their specific access requirements, ensuring that only the necessary portion of data resides in expensive high-performance storage at any given time.
Solution Approach 2:
The patent implements dynamics through automatic data migration between storage tiers based on real-time access frequency monitoring. When data access patterns change, the system dynamically relocates data between high-performance and low-performance storage devices, allowing the storage configuration to adapt continuously to changing workload requirements rather than using a static allocation strategy.
2Loss of energy
If data is arranged in low-performance storage devices, then storage cost is reduced, but response performance deteriorates
Solution Approach 1:
The patent applies preliminary action by proactively migrating data to high-performance storage devices before it is actually needed, based on predicted access patterns. The system analyzes historical access data and workflow characteristics to anticipate which data will be needed soon, moving it to faster storage in advance, thereby preventing performance degradation before it occurs.
Solution Approach 2:
The patent implements feedback mechanisms that continuously monitor data access patterns and use this information to adjust storage placement decisions. The system measures actual access frequencies and timing, compares them against performance targets, and automatically adjusts data migration strategies to maintain optimal response performance while minimizing storage costs.
3Productivity
If hierarchical control is performed based on access frequency, then storage efficiency is improved, but system complexity increases
Solution Approach 1:
The patent applies self-service by enabling the storage system to automatically monitor its own access patterns, analyze data usage characteristics, and perform data migration without external intervention. The hierarchical control mechanism is self-regulating, using built-in monitoring to trigger migrations when access frequencies cross predefined thresholds, eliminating the need for complex external management systems.
Solution Approach 2:
The patent implements parameter changes by using access frequency as the key parameter to determine storage tier placement. The system transforms the complex multidimensional problem of data placement into a simpler decision based primarily on access frequency metrics, allowing hierarchical control to be achieved through monitoring and responding to changes in this single critical parameter rather than managing multiple complex factors simultaneously.
4Speed
If data migration is performed frequently, then response performance is improved, but processing overhead increases
Solution Approach 1:
The patent applies periodic action by performing data migration at regular intervals or when specific access frequency thresholds are reached, rather than continuously monitoring and migrating after every access pattern change. The system uses periodic sampling of access statistics and triggers migrations only when sustained changes in access patterns indicate a need for reconfiguration, reducing the frequency of migration operations while maintaining performance benefits.
Data Source
AI summary
A control apparatus includes a processor configured to execute a procedure including: receiving, from a processing device, a reading request to read a first data piece among a plurality of data pieces included in a data set, the processing device executing a given process on each of the data pieces; reading the first data piece from a first storage tier among a plurality of hierarchical storage tiers having respective different reading capabilities; transmitting the read first data piece to the processing device in response to the reading request; measuring a processing time that the processing device takes to execute the given process; determine a storage tier that is to store the first data piece among the hierarchical storage tiers based on the reading capabilities and the measured processing time; and migrating the first data piece from the first storage tier to the determined storage tier.