Sparkline Cluster Upgrades with Zero-Downtime Query Routing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computing devices experience downtime during updates, such as security patch installations, leading to delayed query execution and reduced user experience.
Innovation Solution
Create a new dataflow interactive cluster as a replacement, register it as an active endpoint, and mark the old cluster as inactive once no active queries exist, allowing seamless transition and minimizing downtime.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a computing device performs updates (security patches, server updates), then system security and efficiency are improved, but downtime occurs causing delayed query execution
Solution Approach 1:
A second dataflow interactive cluster is created in advance before the first cluster is taken offline for updates. This preliminary action ensures that when the first cluster needs to be updated, the second cluster is already ready to take over immediately, eliminating downtime. The second cluster is provisioned with necessary resources and configured to handle query workloads before the transition occurs.
Solution Approach 2:
A gateway node acts as an intermediary between client requests and the dataflow clusters. During the transition period, the gateway node dynamically routes queries to either the first or second cluster based on availability and load. This intermediary enables seamless traffic redirection without requiring clients to be aware of the cluster transition, thus maintaining continuous service availability during updates.
2Productivity
If a computing device performs updates, then system efficiency is improved, but query execution is delayed resulting in lower quality user experience
Solution Approach 1:
The second dataflow interactive cluster is provisioned and configured in advance before updates are applied to the first cluster. This ensures that when users experience any disruption, a fully functional cluster is already available to immediately handle their queries, maintaining user experience quality while allowing necessary updates to occur.
Solution Approach 2:
The system maintains continuous query processing capability by having the second cluster ready to take over from the first cluster. During the transition, the gateway node ensures uninterrupted query execution by routing to the appropriate cluster, thereby maintaining continuous useful action and preventing any break in service that would degrade user experience.
Data Source
AI summary
The present embodiments relate to updating a dataflow interactive cluster with zero downtime. A request to update a first dataflow cluster can be received, and a second dataflow cluster can be generated as a replacement cluster to execute received queries. Generating the second dataflow cluster can include identifying a second series of executor nodes that are configured to execute queries from the gateway node via a second driver node. A first update to a configuration of a host configuration node can be performed to register the second dataflow cluster as an active endpoint and identify the first dataflow cluster as an inactive endpoint. When no active queries exist, a second update to the configuration can be provided to remove the first dataflow cluster from the configuration to direct subsequent queries from the gateway node to the second dataflow cluster.