Auto-scaling Query Engine via Complexity-Based Node Adjustment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern query engines face challenges in scaling big data SQL engine clusters to respond to varying workloads without instability, particularly due to unpredictable CPU usage and the complexity of SQL queries, leading to thrashing and difficulties in draining worker nodes without interrupting in-flight queries.
Innovation Solution
A method for auto-scaling query engines by monitoring query traffic, classifying queries into service classes based on complexity, and adjusting the number of worker nodes based on defined thresholds, with multiple node groups configured for expected durations of queries to minimize the impact on in-flight queries during downscaling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the cluster is scaled using simple host-based metrics such as CPU or memory usage, then the scaling response speed is improved, but the cluster stability deteriorates due to thrashing caused by unpredictable CPU metrics
Solution Approach 1:
The patent changes the scaling parameters from simple host-based metrics (CPU/memory usage) to query-level metrics including query queue depth, active query count, and query class distribution. This parameter transformation resolves the contradiction by providing more stable and meaningful scaling decisions that reflect actual workload needs rather than transient CPU fluctuations.
Solution Approach 2:
The patent implements a feedback mechanism that continuously monitors query traffic patterns, classifies queries by complexity, and adjusts scaling decisions based on observed workload characteristics. This feedback loop prevents thrashing by adapting scaling behavior to actual query execution patterns rather than reacting to volatile CPU metrics.
2Productivity
If the number of worker nodes is increased to handle complex queries, then the query processing capacity is improved, but the cost increases due to provisioning and managing additional nodes
Solution Approach 1:
The patent applies local quality by differentiating worker nodes into multiple query classes (e.g., simple queries, complex queries, heavy queries) and assigning appropriate resources to each class. This allows the system to provision the minimum necessary nodes for each query type rather than uniformly scaling all nodes, optimizing the balance between processing capacity and resource consumption.
Solution Approach 2:
The patent implements partial scaling by selectively adding or removing nodes based on the distribution and characteristics of incoming queries. Rather than scaling the entire cluster uniformly, the system applies partial action to specific query classes, provisioning additional nodes only when and where needed to handle complex or high-volume query workloads.
3Loss of time
If worker nodes are drained quickly during downscaling, then the cost reduction speed is improved, but query interruptions increase due to abrupt node removal
Solution Approach 1:
The patent applies preliminary action by identifying and isolating nodes that can be safely drained before actually removing them from the cluster. The system pre-determines which nodes have completed their current query assignments and are ready for immediate removal, while preserving nodes that are still processing queries. This staged approach enables rapid downscaling of inactive nodes without interrupting active query processing.
Solution Approach 2:
The patent segments the worker node population into distinct groups based on their current workload state (active queries vs. idle/completed queries). This segmentation allows the system to apply different draining strategies to different segments, quickly removing inactive nodes while preserving active ones, thus reducing downscaling time without compromising query reliability.
4Loss of energy
If the cluster is scaled down to reduce costs, then the resource consumption is improved, but the query processing capability deteriorates due to insufficient worker nodes
Solution Approach 1:
The patent implements dynamic scaling that continuously adapts the number of worker nodes to current workload demands. The system monitors query traffic patterns, query class distributions, and processing throughput in real-time, dynamically adjusting cluster size to match actual needs. This dynamic approach prevents both over-provisioning (wasting resources) and under-provisioning (limiting capability), optimizing the balance between resource consumption and processing capability.
Data Source
AI summary
Aspects of the present invention disclose a method, computer program product, and system for auto-scaling a query engine. The method includes one or more processors monitoring query traffic at the query engine. The method further includes one or more processors classifying queries by a plurality of service classes based on a level of complexity of a query. The method further includes one or more processors comparing query traffic for each service class with a concurrency threshold of a maximum number of queries of the service class allowed to be concurrently processed. The method further includes one or more processors instructing auto-scaling of a cluster of worker nodes to change a number of worker nodes available in the cluster based on the comparison, over a defined period of time, of the query traffic relative to a defined upscaling threshold and a defined downscaling threshold.