State Machine Dynamic Distributed Computing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed computing systems execute query processing tasks with static resource allocation and operator execution sequences, which do not adapt to current environmental conditions, leading to inefficiencies and potential failures due to resource disruptions and data distribution skews.
Innovation Solution
A state machine-based approach that dynamically allocates resources and pauses operator execution when service level agreements are violated, allowing for adaptive resource management and in-flight correction, using a pause and play API to manage operator execution and SLA reporting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If static resource allocation and static operator execution sequence are used, then system simplicity is maintained, but system adaptability to current environmental conditions deteriorates
Solution Approach 1:
The patent implements dynamic resource allocation and dynamic operator execution sequencing based on current system conditions. The compiler generates execution plans that can adapt to changing environmental conditions, data distribution skews, and resource disruptions during query processing, resolving the contradiction between system simplicity and adaptability.
2Productivity
If static execution plan is used, then execution predictability is maintained, but execution efficiency under resource disruptions deteriorates
Solution Approach 1:
The system incorporates feedback mechanisms that monitor current system conditions, resource availability, and execution progress. Based on this feedback, the execution plan is dynamically adjusted to maintain efficiency under resource disruptions while ensuring reliable completion of query processing tasks.
Solution Approach 2:
The patent changes execution parameters dynamically based on observed system conditions. When resource disruptions or data distribution skews are detected, the system modifies operator execution sequences, resource allocations, and processing strategies to maintain both efficiency and reliability.
3Adaptability or versatility
If dynamic resource allocation is implemented, then system adaptability improves, but control complexity increases
Solution Approach 1:
The compiler performs preliminary analysis of query workloads and system conditions to pre-generate adaptive execution plans. This preliminary action reduces the complexity of real-time control by preparing multiple execution strategies in advance that can be selected based on observed conditions during query processing.
Data Source
AI summary
A computer readable storage medium with executable instructions specifies the execution of a state machine operating across a set of computing nodes in a distributed computing system. The executable instructions execute a set of operators, where the execution of each operator is under the control of a state machine that periodically invokes pause control states to pause the execution of an operator in response to a violation of a service level agreement specifying an operating condition threshold within the distributed computing system. Partitions of input data are formed that are worked on independently within the distributed computing system. A set of data batches associated with the input data is processed. Data partition control states to process the partitions associated with the set of data batches are specified. Key control states to process a set of keys associated with a data partition of the partitions are defined.