Data Migration Deduplication Storage Utilization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data migration systems in storage centers are inefficient as they solely rely on data relevance without considering storage utilization efficiency, leading to increased costs and storage usage when migrating data from low to high performance systems.
Innovation Solution
A deduplication database is used to predict storage utilization by generating deduplication information for selected datasets, allowing the migration engine to select candidates based on both relevance and deduplication ratios, thereby minimizing storage usage in the high-performance system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data migration is performed based solely on I/O usage profiles to improve data accessibility, then relevant data can be quickly accessed from fast storage, but storage utilization efficiency deteriorates causing increased storage costs
Solution Approach 1:
The system performs preliminary actions by maintaining a deduplication database that stores deduplication information about data in the fast storage array before migration decisions are made. This allows the migration engine to predict deduplication ratios and make informed decisions about which data to migrate, avoiding unnecessary duplication of data already present in fast storage.
Solution Approach 2:
The system implements feedback by using the deduplication database to provide information about existing data in fast storage to the migration engine. The migration engine uses this feedback to identify data with high deduplication ratios (indicating duplicates) and prioritize migration of such data, creating a closed-loop system that continuously optimizes storage utilization.
2Speed
If all data are migrated to high performance storage systems to maximize accessibility, then data access speed is improved, but storage costs increase significantly
Solution Approach 1:
The system changes the parameter used for migration decision-making from solely I/O usage profile to a composite metric that includes deduplication ratio. By incorporating deduplication ratio as a key parameter, the system identifies data that would result in minimal additional storage consumption, thereby reducing the cost of migrating data to high-performance storage while maintaining accessibility for relevant data.
3Productivity
If data with highest I/O usage profile are selected for migration to improve accessibility, then frequently accessed data is prioritized, but storage utilization efficiency deteriorates due to data duplication
Solution Approach 1:
The deduplication database serves as an intermediary between the fast storage array and the migration engine. It contains deduplication information that allows the migration engine to predict how much additional storage space will be consumed by migrating specific data. This intermediary enables the system to balance accessibility needs with storage efficiency by identifying data that can be migrated with minimal additional space requirements.
Data Source
AI summary
Exemplary methods for data migration from a first storage system to a second storage system include maintaining a database containing deduplication information associated with a plurality of remote target storage systems. In one embodiment, the methods include identifying a plurality of storage devices of the first storage system as candidates to be migrated to the second storage system based on access patterns of the storage devices at the first storage system, and selecting one or more storage devices from the identified candidates to be migrated to a second one of the plurality of remote target storage systems that is selected based on the deduplication information of the database. In one embodiment, the methods further include migrating one or more datasets stored at the selected one or more storage devices to the second storage system.