Edge Device Data Migration Using Heat Value Bitmaps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data migration between an edge device and a cloud device requires additional hardware devices, leading to technical complexity and limitations in migrating original data or portions of data, rather than snapshots, which increases storage costs and deployment complexity.
Innovation Solution
A method and device for data migration using a smart Logical Unit Number (LUN) at the edge device that automatically migrates data between edge and cloud memory based on data heat values, eliminating the need for additional hardware and allowing for the migration of original data or portions thereof, by updating data heat values in response to read/write operations and comparing them against a predefined threshold.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If traditional data migration methods are used between edge device and cloud device, then data can be migrated, but additional hardware devices are required leading to increased device complexity and deployment complexity
Solution Approach 1:
The edge device performs data migration autonomously by calculating data heat values itself and making migration decisions without requiring external hardware controllers. The system uses built-in resources (CPU, memory, storage) to implement the migration functionality, eliminating the need for additional dedicated hardware devices while maintaining reliable data migration capability
Solution Approach 2:
The edge device's existing hardware components are made multi-functional by enabling them to perform both normal data processing tasks and data migration operations. The same storage units and processing units that handle regular data operations are also used for calculating heat values and executing migration decisions, thereby eliminating the need for separate dedicated migration hardware
2Quantity of substance
If traditional data migration methods are used, then data can be migrated, but storage costs increase due to inability to migrate original data or portions of data
Solution Approach 1:
The system segments data into data blocks and further divides them into data chunks, enabling granular-level migration control. This segmentation allows the system to migrate only specific portions of data (individual blocks or chunks) rather than requiring migration of entire datasets, thereby optimizing storage capacity utilization and reducing unnecessary storage costs
Solution Approach 2:
The system changes the parameter of data granularity from whole-dataset level to block/chunk level. By introducing configurable data block sizes and chunk divisions, the system can dynamically adjust how much data is migrated based on heat values, enabling precise control over storage capacity utilization and cost optimization
3Speed
If data is stored in edge storage, then access latency is low and performance is high, but storage cost is higher compared to cloud storage
Solution Approach 1:
The system implements dynamic data placement by continuously calculating data heat values and automatically migrating data between edge and cloud storage based on access patterns. Hot data (high heat value) is dynamically placed in edge storage for low-latency access, while cold data (low heat value) is dynamically moved to cloud storage for cost efficiency, creating an adaptive storage system that optimizes both performance and cost
Solution Approach 2:
The system applies different storage quality characteristics to different data blocks based on their heat values. Instead of using a uniform storage strategy for all data, the system places high-priority hot data in high-performance edge storage while placing low-priority cold data in cost-effective cloud storage, optimizing the overall cost-performance balance
4Productivity
If data heat value tracking is implemented using bitmap, then data migration decisions can be made, but memory resources are consumed
Solution Approach 1:
The system segments the bitmap into smaller units corresponding to data blocks and chunks, allowing memory-efficient tracking of heat values. By dividing the large address space into manageable segments that can be processed and stored efficiently, the system reduces overall memory consumption while maintaining the ability to track and make migration decisions for all data
Solution Approach 2:
The system implements partial tracking by focusing bitmap resources on actively monitored data blocks rather than uniformly tracking all data. The bitmap is applied selectively to data blocks that are candidates for migration, allowing the system to achieve effective data migration automation while consuming memory resources only where necessary
Data Source
AI summary
Techniques for data migration involve obtaining, at an edge device, a bitmap in a local memory, the bitmap including a plurality of parameter values. The techniques further involve updating, in response to a data block being written or read, a data heat value of the data block based on the bitmap. The techniques further involve migrating the data block between the edge device and a cloud device based on the updated data heat value. Accordingly, there is a solution for migrating data blocks between an edge device and a cloud device, such that data with a high data heat value is stored at the edge device and data with a low data heat value is stored at the cloud device, thereby improving the performance of a storage system.