Hybrid Statistics Collection for Hadoop Query Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In Hadoop MapReduce environments, traditional statistics collection methods are inadequate due to the unstructured nature of data and reliance on query-executed statistics, which are reactive, limited to simple summaries, not general-purpose, and wasteful in frequency, lacking detailed histograms for accurate optimization.
Innovation Solution
A hybrid statistics collection method that supplements query-only statistics with dedicated collections, using a statistics advisor to identify and rank query fragments that benefit from detailed histograms, allowing proactive scheduling and storage of histograms for improved selectivity estimation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If query-executed statistics collection is used in Hadoop MapReduce environments, then statistics can be gathered without dedicated collection statements, but the statistics are reactive rather than proactive and may be outdated for subsequent queries
Solution Approach 1:
The patent introduces a proactive statistics collection mechanism that gathers statistics in advance of query execution, rather than waiting for queries to trigger collection. The system identifies candidate statistics collection opportunities and schedules them before they are needed, ensuring fresh statistics are available when queries execute. This resolves the contradiction by performing the statistics collection action preliminarily, before the query requires it.
2Reliability
If query-time statistics collection is performed for every job occurrence, then statistics are always available for optimization, but the collection overhead is repeated wastefully when queries are similar
Solution Approach 1:
The patent implements a statistics caching mechanism where statistics collected for one query are stored and reused for subsequent similar queries. The system identifies when queries are semantically similar or identical and reuses the cached statistics from previous executions, avoiding redundant collection operations. This resolves the contradiction by creating a copy of the statistics that can be reused multiple times without additional collection overhead.
Solution Approach 2:
The patent creates general-purpose statistics that can serve multiple different queries, not just the specific query for which they were collected. By collecting statistics at a higher level of abstraction that captures the essential data distribution characteristics, the system makes these statistics universally applicable to various queries operating on the same data, reducing the need for query-specific repeated collection.
3Productivity
If simple summary statistics are collected during MapReduce execution, then collection overhead is low, but complex statistic structures such as histograms cannot be generated
Solution Approach 1:
The patent divides the statistics collection process into separate segments or phases. During MapReduce execution, lightweight summary statistics are collected quickly for immediate use. For queries requiring more detailed information, the system separately initiates targeted histogram collection operations on the relevant data partitions. This segmentation allows the system to maintain high productivity for common cases while providing the option for high measurement precision when needed.
Solution Approach 2:
The patent implements a nested statistics structure where simple summary statistics serve as the outer layer, providing quick overview information. Within this structure, more detailed histogram statistics are nested for specific columns or data partitions that require deeper analysis. The system can traverse this nested structure, using the simple statistics for most queries and drilling down to detailed histograms only when the query optimization requires higher precision.
4Measurement precision
If statistics are collected specifically for given parameter values, then statistics are accurate for those specific queries, but they are not general-purpose for use by a variety of other job occurrences
Solution Approach 1:
The patent transforms query-specific parameterized statistics into general-purpose statistics by changing the parameters from concrete query values to abstract data distribution characteristics. Instead of collecting statistics tied to specific filter constants or query parameters, the system collects statistics about the underlying data distributions, nullity patterns, and value ranges that remain valid across different queries. This parameter transformation maintains measurement precision for the specific query while simultaneously enabling adaptability to other queries with different parameters.
Data Source
AI summary
A system and method for determining optimal query plans within a distributed processing system. The system includes a query optimizer for receiving a query and generating a query plan for execution by the distributed processing system; a repository including statistics related to prior executed query plans; a plan execution engine for receiving and executing the query plan, collecting statistics corresponding to the query plan, and storing the statistics and optimizer usage metadata corresponding to the query plan and prior executed query plans in the repository. The system further includes a statistics advisory tool for mining the metadata to generate a collection of histogram statistics and storing said histogram statistics in the repository. When generating a query plan, the query optimizer searches the statistics repository for statistics, including histogram statistics, useful to the generation of the current query plan and uses selected statistics from the repository to generate the query plan.