Parallel Data Transfer via Chunked Encryption
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data transfer methods are limited by protocols like TCP, processing capabilities, and resource utilization, leading to inefficiencies in data transmission and potential bottlenecks in receiving systems, particularly due to single-server limitations and resource underutilization.
Innovation Solution
Implementing parallelization techniques such as deconstructing data objects into chunks for simultaneous transmission across multiple servers, separate encryption of each chunk using a cryptographic key with initialization vectors, and utilizing multi-part upload/download capabilities to enhance transfer efficiency and reliability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is transferred using conventional single-server methods, then the transfer process is simple to implement, but the throughput is limited and resource utilization is unbalanced
Solution Approach 1:
The patent divides a data object into multiple chunks and distributes them across multiple servers for parallel transfer. Each chunk is independently encrypted and transmitted, allowing simultaneous processing by multiple servers. This segmentation increases throughput by utilizing multiple communication channels while maintaining manageable system complexity through modular chunk handling.
Solution Approach 2:
The patent transitions from single-server sequential transfer to multi-server parallel transfer by adding the server dimension. Multiple servers operate simultaneously on different data chunks, transforming the transfer process from a one-dimensional sequential operation to a multi-dimensional parallel operation, thereby significantly improving throughput.
2Reliability
If encryption is performed on the entire data object before transfer, then security is maintained, but the processing time increases the transfer duration
Solution Approach 1:
The patent encrypts each data chunk independently rather than encrypting the entire data object as a single unit. This allows encryption operations to be performed in parallel on multiple chunks simultaneously, reducing the total encryption processing time while maintaining the same security level through consistent cryptographic application on each segment.
Solution Approach 2:
The patent performs encryption on data chunks before they are transmitted to servers. By pre-encrypting chunks locally on the client side before upload, the system eliminates the need for server-side decryption and re-encryption operations, thereby reducing overall processing time while maintaining security throughout the transfer process.
3Reliability
If TCP protocol is used for data transfer, then reliable delivery is ensured, but the acknowledgment requirement creates time delays
Solution Approach 1:
The patent divides data into chunks and transfers them independently through multiple servers. This segmentation allows parallel transmission of multiple chunks without requiring sequential acknowledgments for each complete data object, effectively reducing the impact of TCP acknowledgment delays on overall transfer time while maintaining reliable delivery through chunk-level error handling.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Data transfer between a first computer system and a second computer system utilize parallel servers of the second computer system. A plurality of data chunks collectively comprise a data object. The data chunks may be encrypted and sent over parallel channels to the second computer system, which may be a data storage service of a computing resource service provider. The data object, or a portion thereof, may be downloaded from the data storage system in parallel.