Engine-Level Data Model Optimization Using Reusable Hash Plans
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing analytics systems lack the ability to dynamically optimize database operations at an engine-by-engine level, leading to performance issues due to non-granular and static optimizations, which are not customer-specific and can cause degradation.
Innovation Solution
A mechanism is provided to dynamically determine optimizations at runtime for each engine in the database, using a hash representation of the data model to reuse past optimizations and generate customer-specific engine-level optimizations, with the option of machine learning for automatic optimization generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If static optimizations are applied at server level, then implementation simplicity is maintained, but performance degradation occurs due to lack of granularity and customer-specific adaptation
Solution Approach 1:
The patent segments the monolithic server-level optimization into granular engine-level optimizations. Each database engine (analytical engine, MDS engine, calc engine, SQL engine) receives tailored optimizations specific to its processing characteristics and the customer's data model, enabling precise performance tuning without overwhelming system-wide complexity.
Solution Approach 2:
The system dynamically determines optimizations at runtime based on the specific data model, customer requirements, and engine state. Rather than applying static server-level optimizations, the system adapts optimization strategies for each engine based on real-time conditions, improving performance while managing complexity through automated dynamic adjustment.
2Productivity
If engine-level optimizations are generated dynamically, then customer-specific performance is improved, but computational overhead increases due to runtime analysis and optimization generation
Solution Approach 1:
The system performs preliminary actions by analyzing the data model structure and generating optimization strategies before query execution. By determining engine-level optimizations in advance based on the data model characteristics, the system prepares optimized execution plans that can be applied immediately, reducing runtime overhead while maintaining customer-specific performance.
Solution Approach 2:
The system creates optimized copies of query execution plans for each engine based on the original data model analysis. Rather than re-analyzing during execution, the system generates and stores optimized versions that can be reused, minimizing computational overhead while delivering tailored performance for each customer's specific data patterns.
3Productivity
If optimizations are applied without hashing, then all data models can be processed, but duplication of optimizations occurs leading to wasted resources
Solution Approach 1:
The system changes the parameter representation of data models by generating hash values from their structure. This transformation enables efficient comparison and identification of duplicate data models, allowing the system to recognize when the same optimization has already been generated and avoid redundant CPU-intensive optimization processes, thereby reducing energy consumption while maintaining comprehensive coverage.
4Ease of operation
If machine learning is used for automatic optimization generation, then ease of operation is improved, but system complexity increases due to ML model integration
Solution Approach 1:
The system enables self-service by implementing machine learning models that automatically generate and apply engine-level optimizations without requiring manual intervention from customers or administrators. The ML models autonomously analyze data models, determine optimal optimization strategies for each engine, and apply them automatically, improving ease of operation while managing complexity through automated self-configuration.
Data Source
AI summary
In an example embodiment, a mechanism is provided to dynamically determine optimizations of data model execution at runtime at an engine-by-engine level, meaning that one optimization may be applied to one engine and not others. This mechanism is enabled by also providing a method by which model optimizations are stored and retrieved using a hash representation of each model, allowing past optimizations to be reused. Thus, the next time a similar model is built, optimizations of an earlier model can be reused. A definition store is also used as a basis to create new optimizations dynamically.