Portable Storage Data Migration via Manifest Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Migrating large amounts of data objects from a distributed data store to another destination is time-consuming and costly, and existing methods of using portable storage devices are inefficient, especially when dealing with changing or new data objects, leading to errors and increased computing resources usage.
Innovation Solution
Generating an inventory of data objects, dividing it into smaller manifests, and assigning each to a portable storage device, allowing for parallel and asynchronous copying, which reduces migration time and processing costs, and enables easier handling of changes during the process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If data objects are sequentially transferred to portable storage devices, then data migration can be performed, but migration time and computing costs increase significantly
Solution Approach 1:
The patent divides the large dataset into multiple smaller manifests, each assigned to a separate portable storage device. This segmentation enables parallel processing of multiple data subsets simultaneously, dramatically reducing total migration time compared to sequential transfer of a single large dataset.
Solution Approach 2:
The system generates an inventory and divides it into manifests before the actual data transfer begins. This preliminary organization allows multiple portable storage devices to be prepared and dispatched simultaneously, enabling parallel migration operations from the outset rather than sequentially after deployment.
2Quantity of substance
If data is split across multiple portable storage devices, then large datasets can be migrated, but handling changing or new data objects becomes difficult and error-prone
Solution Approach 1:
The patent implements a feedback mechanism where the system continuously monitors for new or changed data objects during the migration process. When changes are detected, the system can update the manifests and reassign data objects to appropriate portable storage devices, ensuring data consistency across all devices even as the source data evolves.
Solution Approach 2:
The manifest assignment system is designed to be dynamic rather than static. Manifests can be updated and reassigned during the migration process to accommodate new or changed data objects, allowing the system to adapt to changing data conditions while maintaining reliability across multiple portable storage devices.
3Ease of manufacture
If sequential key/value pair listing is used to split data objects, then data can be organized for transfer, but the process becomes extremely slow and inefficient
Solution Approach 1:
Instead of creating one large sequential listing of all key/value pairs, the patent segments the inventory into multiple smaller manifests that can be processed and transferred in parallel. This segmentation maintains the organizational benefits of listing while eliminating the sequential bottleneck by allowing simultaneous processing of multiple manifest subsets.
Data Source
AI summary
Technologies are disclosed for migrating large amounts of data objects, e.g., billions of data objects, from a distributed data store to another data store or computing device. A method includes generating an inventory of the data objects in the distributed data store and dividing the inventory into multiple listings of data objects. Each data object is identified in only one corresponding listing. Manifests are generated based upon the listings. Each manifest is assigned to a corresponding portable storage device of a plurality of portable storage devices, and the data objects and listings are transferred, based at least in part, upon the manifests to the plurality of portable storage devices. The inventory of the data objects and the plurality of portable storage devices are provided to an entity associated with the another data store or computing device. The data objects are transferred from the plurality of portable storage devices to the another data store or computing device using the inventory.