Query Processing Engine Recommendation via Feature Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Organizations face challenges in efficiently executing queries on massive data sets due to the limitations of existing massively parallel query processing frameworks like Hadoop, which are slow for interactive analytics and require optimization for specific data types and hardware configurations, making it difficult to evaluate and select the best query processing engine for varying queries.
Innovation Solution
A platform that automatically selects and executes queries on the most suitable query processing engine by extracting query features, comparing them to an evolving knowledge base, and using machine learning to update and recommend the best engine based on service level agreement requirements and execution statistics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If MapReduce is used for data processing on Hadoop, then storage capacity is improved, but query response time deteriorates
Solution Approach 1:
The patent creates a universal query processing platform that can execute the same query across multiple different query processing engines (Hive, Impala, Spark, Presto) simultaneously. This multi-functionality allows the system to leverage the storage capacity of Hadoop while overcoming the slow query response time by selecting the most appropriate engine for each query type.
Solution Approach 2:
The system dynamically changes the execution parameters by selecting different query processing engines based on query features, data characteristics, and performance requirements. The recommendation engine analyzes query parameters and transforms them into engine selection decisions, effectively changing the execution environment to optimize both storage utilization and query response time.
2Productivity
If query processing engines are optimized for specific data types and formats, then processing efficiency is improved, but adaptability deteriorates
Solution Approach 1:
The patent implements a universal query processing platform that maintains connections to multiple specialized query processing engines, each optimized for specific data types and formats. The system's adaptability is achieved through its ability to route queries to the most appropriate specialized engine based on the query features and data characteristics, thus combining the processing efficiency of specialized engines with the adaptability of a universal interface.
Solution Approach 2:
The recommendation engine acts as an intermediary between the user's query and the multiple specialized query processing engines. It analyzes the query features, data characteristics, and engine capabilities to mediate the selection of the most appropriate engine, thereby enabling specialized engines to maintain their optimization for specific data types while the overall system remains highly adaptable to various query types.
3Adaptability or versatility
If multiple query processing engines are available, then query execution flexibility is improved, but system complexity deteriorates
Solution Approach 1:
The patent extracts the complexity of engine selection and performance optimization from the user's responsibility by implementing an automated recommendation engine. This engine analyzes query features, evaluates multiple query processing engines, and recommends the best execution option, thereby maintaining query execution flexibility while reducing the operational complexity for users and administrators.
Solution Approach 2:
The system implements feedback mechanisms where execution statistics and performance metrics from multiple query processing engines are continuously collected and analyzed. This feedback is used by the recommendation engine to learn from past performance and improve future engine selections, thereby managing system complexity through data-driven automation while maintaining high query execution flexibility.
4Ease of operation
If HiveQL is used for SQL-like querying, then ease of use is improved, but query speed deteriorates
Solution Approach 1:
The recommendation engine serves as an intermediary that accepts HiveQL queries from users and automatically determines the most appropriate query processing engine for execution. This allows users to continue using the easy-to-write HiveQL syntax while the system translates and executes the query on faster engines like Impala, Spark, or Presto when appropriate, thereby maintaining ease of use while improving query speed.
Solution Approach 2:
The system changes the execution parameter of the query by selecting different engines based on the query characteristics. When a HiveQL query is submitted, the recommendation engine analyzes its features and may redirect execution to a faster engine, effectively changing the speed parameter while maintaining the same user-friendly HiveQL interface for the user.
Data Source
AI summary
Methods and systems are disclosed for providing recommendation of a query processing engine from multiple query processing engines. The method includes receiving queries by a query receiving unit, extraction of features of the query by a feature extractor, checking for a similar query in a knowledge base by an evaluator, executing the query on multiple query processing engines by a scheduler, and collecting execution statistics for each query processing engine if the similar query is not found in the knowledge base or fetching the suitable query processing engine from the knowledge base if the similar query is found in the knowledge base and executing the query on that query processing engine. The method also includes providing the recommendation of a single query processing engine from multiple query processing engines by the evaluator based on a recommendation value calculated for every query processing engine.