Data Transfer Concurrency Control via Queue-Based Throttling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems for large-scale data transfer face challenges such as concurrency limitations, throttling, and write capacity issues, which can lead to inefficiencies and require manual intervention or additional costs to manage bandwidth and resources.
Innovation Solution
The system employs a processor and memory with instructions to manage concurrency by receiving notifications from a data stream, passing messages to a queue with a message group ID, and dynamically controlling the number of invocations and routines to write data to a cloud-based database, thereby mitigating throttling and write capacity limits.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If large amounts of data are transferred from a data warehouse to a cloud-based database, then the data accessibility for external applications is improved, but concurrency limitations and throttling occur that slow or stop the data-transfer process
Solution Approach 1:
The patent segments the data transfer process into multiple concurrent data transfer operations, each handling a portion of the total data. This allows the system to bypass concurrency limitations by distributing the transfer load across multiple parallel operations, thereby maintaining high throughput while avoiding throttling that would occur with a single large-scale transfer operation.
Solution Approach 2:
The system dynamically adjusts the number and configuration of concurrent data transfer operations based on system conditions, database capacity, and observed throttling patterns. This dynamic adaptation allows the data transfer process to respond to changing conditions in real-time, maintaining reliability while optimizing throughput by scaling the number of concurrent operations up or down as needed.
2Productivity
If concurrency is increased to speed up data transfer, then productivity is improved, but throttling and write capacity limits are exceeded causing the process to fail
Solution Approach 1:
The system implements feedback mechanisms that monitor database write capacity, throttling conditions, and the performance of concurrent data transfer operations. Based on this feedback, the system automatically adjusts the number of concurrent operations and their configuration to maintain optimal transfer speed while staying within database capacity limits and avoiding throttling, thus resolving the contradiction between speed and capacity constraints.
Solution Approach 2:
The patent changes key parameters of the data transfer process, including the number of concurrent operations, batch sizes, and retry intervals, to optimize performance. By dynamically adjusting these parameters based on system conditions, the system achieves high data transfer speed without exceeding write capacity limits or triggering throttling mechanisms.
3Ease of operation
If manual intervention is used to manage data transfer, then control over the process is improved, but the complexity of operation increases and human intervention is required
Solution Approach 1:
The system implements self-service capabilities where the data transfer process automatically manages its own concurrency levels, monitors database capacity, and adjusts operations without human intervention. The system autonomously handles throttling conditions, retries failed operations, and optimizes transfer parameters, thereby maintaining ease of operation through automation while reducing the need for manual control.
Solution Approach 2:
Through feedback loops that continuously monitor system conditions and transfer performance, the system automatically adjusts its operation to maintain optimal control. This feedback-driven automation enables the system to manage complex concurrency and capacity issues without requiring human intervention, while still providing controlled and predictable data transfer behavior.
Data Source
AI summary
Systems and methods for fine-tuned control over data transfer processes. An exemplary data transfer process may include: receiving a data stream at a storage service; receiving, at a first function, one or more notifications; in response to each notification, passing, by the first function, a message to a queue, the message comprising an address of a respective file within the storage service; receiving, at an invocation of a second function at a second computing service, one or more messages from the queue; retrieving, by the second function, data from one or more files based on the address in each of the one or more messages; and writing, by the second function, the data to a database. Systems and methods according to aspects of the present disclosure improve processes of transferring data from a data warehouse or database to a cloud-based database by mitigating back-pressure, auto-scaling throughput, and controlling concurrency scaling.