Data Characteristic Tuning for Heterogeneous Query Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Big data systems with heterogeneous storage mechanisms lack support for traditional cost-based query tuning due to the absence of features like indexes and data cardinality statistics, hindering efficient data storage and retrieval, particularly in time-series data environments.
Innovation Solution
A system that categorizes data into three families - Identifier, Usecase, and Runtime Data - using machine learning to deduce storage and retrieval patterns, optimizing query execution by grouping data based on inherent aspects and implied patterns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional cost-based query tuning is used, then query optimization is possible, but heterogeneous storage mechanisms lack the necessary features (indexes, cardinality statistics) to support it
Solution Approach 1:
The system performs self-service by automatically collecting runtime data from queries, analyzing access patterns, and generating optimization rules without requiring traditional database features. The query tuner observes actual query behavior and autonomously creates indexing and optimization strategies tailored to the specific workload and storage characteristics.
Solution Approach 2:
The patent introduces an intermediary layer (the query tuner system) between the heterogeneous storage mechanisms and the query processing engine. This intermediary collects runtime statistics, analyzes query patterns, and translates them into optimization rules that bridge the gap between the storage system's limitations and the query optimizer's needs.
2Adaptability or versatility
If data is stored in heterogeneous storage mechanisms, then flexibility and scalability are improved, but traditional query tuning features like indexes and cardinality statistics are lost
Solution Approach 1:
The system performs preliminary action by proactively collecting and analyzing runtime query data before optimization is needed. It continuously monitors access patterns, builds statistical models of data distribution and query behavior, and pre-generates optimization rules that can be quickly applied when queries are executed.
Solution Approach 2:
The patent implements feedback loops where runtime query performance data is collected, analyzed, and used to refine optimization rules. The system continuously monitors actual query execution, compares predicted vs. actual performance, and adjusts its statistical models and optimization strategies based on this feedback to improve over time.
3Measurement precision
If machine learning is used to deduce data patterns, then query optimization accuracy is improved, but system complexity increases
Solution Approach 1:
The system uses lightweight, simple machine learning models that can be quickly trained and discarded rather than complex models requiring extensive computational resources. The approach favors simple statistical analysis and pattern recognition over sophisticated ML algorithms, using computationally inexpensive methods that provide sufficient accuracy for the task.
Data Source
AI summary
Techniques for improving system performance based on data characteristics are disclosed. A system may receive updates to a first data set at a first frequency. The system selects a first storage configuration, from a plurality of storage configurations, for storing the first data set based on the first frequency, and stores the first data set in accordance with the first storage configuration. The system may further receive updates to a second data set at a second frequency. The system selects a second storage configuration, from the plurality of storage configurations, for storing the second data set based on the second frequency, and stores the second data set in accordance with the second storage configuration. The second storage configuration is different than the first storage configuration.