Chunked Data Transfer with Upsert and Retry Logic
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data transfer methods face challenges in ensuring complete and error-free transfer of large datasets from a source data store to a destination data store, particularly when the formats differ, leading to incomplete transfers or failures due to network issues and format incompatibilities.
Innovation Solution
The system employs advanced retry mechanisms by splitting datasets into chunks, performing upsert operations to convert them into a compatible format, and using separate engines for encryption and decryption, with a failed chunk retry mechanism that re-attempts transfers using a back-off retry scheme to ensure lossless data transfer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is transferred in large batches without retry mechanisms, then transfer speed is improved, but data integrity deteriorates due to network failures and format incompatibilities
Solution Approach 1:
The patent divides large datasets into smaller chunks before transfer. Each chunk is independently processed and can be retried separately if it fails, maintaining both transfer efficiency and data integrity through granular error handling.
Solution Approach 2:
The system performs format conversion and validation operations on data chunks before actual transfer. This preliminary preparation ensures that data is ready for the destination schema, reducing transfer failures and enabling faster reliable transfers.
2Adaptability or versatility
If data format conversion is performed during transfer, then compatibility with destination schema is improved, but transfer time increases due to additional processing
Solution Approach 1:
Format conversion is performed on individual data chunks rather than entire datasets. This segmentation allows parallel processing of conversion and transfer operations, reducing total processing time while maintaining schema compatibility.
Solution Approach 2:
Data chunks are converted to the destination schema format before transfer begins. This preliminary conversion eliminates format incompatibility issues during transfer and allows for more efficient data movement.
3Reliability
If failed data chunks are immediately retried without back-off, then data integrity is improved through rapid retransmission, but system stability deteriorates due to network overload
Solution Approach 1:
The system implements periodic retry attempts with exponential back-off intervals. Failed chunks are retried at increasing time intervals (e.g., 1 second, 2 seconds, 4 seconds), allowing the network to stabilize while ensuring eventual delivery of all data chunks.
Solution Approach 2:
The retry mechanism includes built-in time buffers and back-off periods that cushion the system against network instability. These pre-planned delays prevent overwhelming the network while ensuring data is eventually transmitted successfully.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
In non-limiting examples of the present disclosure, systems, methods and devices for transmitting data stored in a source data store to a destination data store are presented. A plurality of data chunks may be received in a first format. At least one upsert operation may be performed on each of the data chunks for converting the first format to a format compatible with a destination data store schema. A transfer of the upserted data chunks to the destination data store may be executed. A determination may be made that at least one of the upserted data chunks has failed to transfer to the destination data store. Failed data chunks may be stored in a failed chunk retry data store. The at least one upsert operation may be re-performed on failed data chunks and a re-execution of the transfer of failed data chunks to the destination data store may be made.