Cloud Data Migration via Fingerprint Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional methods for migrating data from one cloud storage service to another are disruptive, time-consuming, and inefficient, often requiring cloud services to be taken offline or involving costly temporary storage of petabytes of data.
Innovation Solution
A system and method for online, non-disruptive data migration across cloud providers, utilizing a cloud migration module that identifies live and dead segments, redirects application reads and writes, and employs a Perfect Hash Vector for efficient data transfer without recalling data to local storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional data migration methods are used (taking cloud service offline or recalling data to local storage), then data can be transferred to a new cloud storage service, but application access to data is interrupted and storage costs increase
Solution Approach 1:
The system performs preliminary actions by establishing a fingerprint index and metadata structure before migration begins. This allows the system to identify and track live data segments that need to be migrated, enabling continuous access during the migration process without interrupting applications.
Solution Approach 2:
The patent introduces an intermediary migration system that acts as a mediator between the source and destination cloud storage services. This intermediary manages the migration process by redirecting application reads and writes, coordinating data transfer, and maintaining access continuity without requiring applications to be aware of the migration.
2Ease of manufacture
If data is recalled from cloud storage to local storage for transfer, then data can be migrated to a new cloud, but storage costs and time consumption increase significantly
Solution Approach 1:
The system extracts only the necessary metadata (fingerprint index and container references) from the source cloud storage, rather than extracting and transferring the actual data. This allows the system to identify live data segments and migrate them directly between cloud services without recalling data to local storage, significantly reducing storage costs and energy consumption.
Solution Approach 2:
The patent uses copying by creating and maintaining a fingerprint index that references data container locations. Instead of physically moving all data, the system copies metadata references and uses this index to redirect access to migrated data, enabling efficient data location tracking without expensive data recall and re-transfer.
3Loss of information
If all data segments are migrated to ensure completeness, then no data is lost, but network bandwidth and time are wasted transferring dead segments
Solution Approach 1:
The system applies local quality by differentiating between live and dead data segments using the fingerprint index. Each data segment is evaluated individually to determine if it is still referenced by active applications (live) or can be discarded (dead). This allows selective migration of only necessary data, improving efficiency while maintaining completeness of live data.
Solution Approach 2:
The patent implements partial action by migrating only the necessary subset of data segments that are still in use (live segments). Instead of performing excessive action by migrating all segments including dead ones, the system uses the fingerprint index to identify and transfer only the required portions, significantly improving migration efficiency while ensuring no live data is lost.
Data Source
AI summary
An index associates fingerprints of file segments to container numbers of containers within which the file segments are stored. At a start of migration, a boundary is created identifying a current container number. At least a subset of file segments at a source storage tier are packed into a new container to be written to a destination storage tier. A new container number is generated for the new container. The index is updated to associate fingerprints of the at least subset of file segments to the new container number. A request is received to read a file segment. The index is queried with a fingerprint of the file segment to determine whether the request should be directed to the source or destination storage tier based on a container number of a container within which the file segment is stored.