Remote Data Migration With Delta Extraction and Version Sets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for migrating data from on-premises to remote computing environments are inefficient, inaccurate, and costly, particularly when dealing with heterogeneous data from various applications.
Innovation Solution
A framework that identifies and transmits only data changes, standardizes data formats, and segregates data into current, previous, and invalid representations to optimize data transfer, using a device with a processor, communications module, and memory to manage data sets and schema updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all data is transmitted from on-premises to remote computing environment, then data migration completeness is improved, but bandwidth usage and transmission time increase significantly
Solution Approach 1:
The patent extracts and transmits only the changed portions of data (deltas) rather than complete data sets. The system identifies and extracts only modified records, columns, or blocks between source and destination, significantly reducing transmission volume while maintaining data completeness for updated information.
Solution Approach 2:
The patent implements partial data transmission by sending only necessary data changes rather than complete data sets. The system determines the minimal required transmission scope based on change detection, avoiding excessive transmission of unchanged data while ensuring all necessary updates are captured.
2Manufacturing precision
If data is standardized and processed on on-premises systems before upload, then data quality and consistency improve, but processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary data standardization and validation on the source system before data extraction. Schema validation, data type checking, and format standardization are executed upfront, ensuring data quality is established before transmission begins, reducing reprocessing needs at the destination.
Solution Approach 2:
The patent introduces an intermediary data extraction and transformation layer that sits between the source and destination systems. This intermediary component handles standardization and validation tasks, acting as a buffer that prepares data for efficient transmission without requiring extensive processing at either endpoint.
3Stability of the object's composition
If complete data sets are uploaded including unchanged data, then data consistency is ensured, but transmission bandwidth and storage costs increase
Solution Approach 1:
The patent extracts only changed data elements (rows, columns, or blocks) between source and destination systems. By comparing data states and identifying modifications, the system extracts minimal necessary data for transmission, maintaining consistency for updated elements while eliminating redundant unchanged data from the transmission stream.
4Adaptability or versatility
If data migration framework handles heterogeneous data from various applications, then system versatility improves, but framework complexity and implementation difficulty increase
Solution Approach 1:
The patent implements a universal data extraction framework that can handle multiple data sources, formats, and applications through a single unified interface. The system provides multi-functional capabilities to extract, transform, and load data from diverse sources using common protocols and connectors, reducing the need for application-specific custom implementations.
Data Source
AI summary
A device, method and system for loading data into a remote computing environment is disclosed. The method includes receiving a request to load a new data set into a remote computing environment, the new data set impacting a data set stored thereon. The method includes identifying one or more changes to a current representation of the data set within the new data set, the one or more changes replacing information in the current representation. The method includes transmitting the identified one or more changes to a data store persisting the current representation. The method includes transmitting the replaced information to a data store persisting a previous representation of the data set. The method includes transmitting other information in the new data set that is determined to be invalid data to a data store persisting an invalid data set associated with the data set.