Learned Query Cost Modeling for Big Data Plan Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cost modeling in big data systems is challenging due to complexity and variance in cloud environments, leading to inaccurate cost estimates and suboptimal query execution plans, which are difficult to tune consistently across workloads and over time, especially in dynamically provisioned cloud resources.
Innovation Solution
A system that evaluates resource consumption of queries by transforming logical operator representations into multiple physical operator representations, applying trained resource consumption models with varying granularity and accuracy, and selecting the most efficient plan based on these estimates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional cost models are used in big data systems, then device complexity is reduced, but measurement precision of resource consumption deteriorates
Solution Approach 1:
The patent segments the cost modeling problem by creating separate resource consumption models for different physical operators (scan, filter, join, aggregate, etc.). Each model is trained independently on historical execution data specific to that operator type, allowing for specialized accurate predictions without requiring a single monolithic complex model. This segmentation enables the system to achieve high measurement precision across diverse query operations while managing complexity through modular model design.
2Measurement precision
If cost models are tuned for specific workloads, then measurement precision improves, but adaptability to different workloads deteriorates
Solution Approach 1:
The patent employs parameter changes by training each resource consumption model with adjustable parameters that can be dynamically tuned based on the specific workload characteristics. The models learn from historical execution data and can adapt their parameters to different query patterns, data distributions, and system configurations. This allows the same model framework to maintain high accuracy across diverse workloads without requiring complete re-tuning, achieving both precision and adaptability.
3Measurement precision
If multiple resource consumption models are applied to physical operator representations, then measurement precision improves, but device complexity increases
Solution Approach 1:
The patent segments the model application process by creating a dedicated resource consumption evaluator component that systematically applies multiple specialized models to different physical operators in the query plan. Each operator type (scan, filter, join, etc.) has its corresponding trained model applied, and the results are aggregated to produce the overall cost estimate. This segmented approach manages complexity through structured organization while leveraging the combined accuracy of multiple models.
Solution Approach 2:
The patent implements a universal resource consumption evaluator framework that can apply multiple different models in a consistent manner across all query plans. This multi-functional evaluator handles various operator types, model formats, and aggregation methods through a single unified interface, reducing the operational complexity despite using multiple underlying models. The evaluator serves multiple functions: model selection, parameter passing, result aggregation, and plan comparison.
Data Source
AI summary
Methods, systems, apparatuses, and computer program products are provided for evaluating a resource consumption of a query. A logical operator representation of a query generated to be executed (e.g., obtained from a query generating entity) may be determined. The logical operator representation may be transformed to a plurality of different physical operator representations for executing the query. A plurality of resource consumption models may be applied to each of the physical operator representations to determine a resource consumption estimate for the physical operator representation. The resource consumption models may be trained in different manners based at least on a history of query executions, such that each model may have different granularity, coverage and/or accuracy characteristics in estimating a resource consumption of a query. Based on the determined resource consumption estimates for the physical operator representations, a particular one of the physical operator representations may be selected to execute the query.