Concurrent Dataflow Execution with Write Conflict Protection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large-scale cloud-based computing systems, concurrent execution of dataflows leads to a risk of write conflicts, which can result in record corruption and inaccurate query results, and existing solutions like serialization or queuing create bottlenecks and latency issues, especially in high-demand environments serving millions of users.
Innovation Solution
Implementing a system that creates dataflow definitions, generates dataflow versions, and uses a message queue to select runnable dataflows based on system limits and write conflict verification, allowing concurrent execution while ensuring data integrity through optimistic locking and FIFO ordering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If dataflows are serialized or queued to prevent write conflicts, then data integrity is maintained, but system latency increases and throughput decreases
Solution Approach 1:
The system dynamically adjusts concurrency levels based on write conflict detection. Multiple dataflows of the same type can execute concurrently when no write conflicts are detected, but automatically serialize when conflicts are identified, allowing the system to adapt its execution strategy in real-time rather than using fixed serialization
Solution Approach 2:
The patent changes the parameter of execution mode from always-serialized to conditionally-concurrent based on write conflict status. By monitoring whether write conflicts occur and adjusting the concurrency parameter accordingly, the system achieves both high throughput (when safe) and data integrity (when conflicts are detected)
2Reliability
If dataflows are serialized or queued to prevent write conflicts, then accurate query results are ensured, but system throughput decreases
Solution Approach 1:
The system dynamically switches between concurrent and serialized execution modes based on write conflict detection, allowing high throughput when no conflicts occur while maintaining query accuracy through automatic serialization when conflicts are detected
Solution Approach 2:
The patent implements a feedback mechanism where write conflict detection results are used to control future dataflow execution. When write conflicts are detected, the system feeds back this information to serialize subsequent dataflows of the same type, ensuring query accuracy while minimizing unnecessary serialization
3Productivity
If multiple dataflows of the same type execute concurrently, then system throughput increases, but write conflicts occur leading to record corruption
Solution Approach 1:
The patent introduces an intermediary mechanism (write conflict detection and control logic) that mediates between concurrent dataflows. This intermediary monitors for write conflicts and coordinates execution to prevent corruption while allowing concurrent throughput when safe
Solution Approach 2:
The system replaces the mechanical serialization approach with a smarter control mechanism that uses write conflict detection and conditional execution. This substitutes brute-force sequential processing with a more sophisticated system that allows concurrency when safe but prevents corruption when risks are detected
Data Source
AI summary
In accordance with disclosed embodiments, there are provided systems, methods, and apparatuses for implementing concurrent dataflow execution with write conflict protection within a cloud based computing environment. For instance, an exemplary system having at least a processor and a memory therein includes means for: creating a dataflow definition for a first dataflow type, wherein the dataflow definition includes at least one or more datasets to be accessed by the dataflow and a plurality of functional operations to be performed on the one or more datasets when the dataflow is executed; generating and storing a dataflow version identifying all datasets accessed by the dataflow based on the dataflow definition created; receiving multiple requests for the first dataflow type; enqueuing the multiple requests into a message queue pending execution; selecting, from the message queue, a first runnable dataflow having been earliest enqueued of the first dataflow type for execution based on (i) the first dataflow type being allowable within system limits and based further on (ii) verification that the selected first runnable dataflow is not already executing and based further on (iii) verification there is no write conflict for any dataset accessed by the selected first runnable dataflow. Other related embodiments are disclosed.