Date-Driven Data Movement for Petabyte-Scale Cloud Migration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data movement approaches are constrained when dealing with large data volumes, particularly in migrating data from on-premise systems to the cloud, often requiring physical devices and manual specification of files, which is inefficient and prone to errors.
Innovation Solution
A data movement framework that operates at the table level, using a table-based characterization and date specification to automatically determine and transfer files, reducing the need for manual file listing and enabling efficient, secure, and scalable data migration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If physical data boxes are used to move large amounts of data, then data transfer capability is improved, but device complexity and manual operation requirements increase
Solution Approach 1:
The patent replaces the mechanical physical data box system with a software-based automated data movement framework. Instead of using physical boxes that require manual connection and file specification, the system uses automated software agents that can identify and transfer data based on logical criteria such as date ranges and file patterns, eliminating the need for physical infrastructure and manual operations.
Solution Approach 2:
The data movement system performs self-service by automatically identifying files to be transferred based on specified criteria (date ranges, file patterns) without requiring manual file listing. The system autonomously determines which data needs to be moved, executes the transfer, and manages the process end-to-end, reducing human intervention to simple parameter specification.
2Ease of operation
If manual file specification is used for data transfer, then data movement can be performed, but time consumption and error probability increase
Solution Approach 1:
The system enables self-service data transfer by automatically generating the list of files to be transferred based on user-specified criteria such as date ranges and file patterns. Instead of requiring manual file enumeration, the software autonomously identifies matching files and prepares them for transfer, dramatically reducing the time and effort required for data movement operations.
Solution Approach 2:
The system performs preliminary actions by pre-defining transfer criteria (date ranges, file patterns, source and destination paths) that enable automatic file identification before the actual data transfer begins. This preliminary configuration allows the system to autonomously determine which files need to be moved without requiring manual intervention during the transfer process.
3Quantity of substance
If existing data transfer approaches are used for 1 PB scale data, then data movement is possible, but processing efficiency and resource utilization worsen
Solution Approach 1:
The patent applies segmentation by dividing the large-scale data transfer process into manageable components: the system specifies data to be moved at the table level rather than individual file level, automatically determines the set of files to be transferred, and initiates transfer in an organized manner. This segmentation approach improves processing efficiency and resource utilization when handling 1 PB scale data migrations.
Data Source
AI summary
Computing platforms, methods, and storage media for data movement are disclosed. Exemplary implementations may: obtain a data transfer command including a date specification of a set of data to be transferred; automatically determine, based on the date specification of the data to be transferred, a set of files to be transferred; and initiate transfer of the set of files. In an implementation, data is moved based on the date specification, and the files associated with the specified date range are automatically determined. A single command, which may reference a CSV file, may be used to efficiently and reliably transfer a large amount of data without a user having to specify the specific files to be transferred.