Query Orchestrator Calculation Graph for Multi-Source Data Blending
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face difficulties in processing queries that blend data from multiple sources efficiently, as they require complex data retrieval and blending operations that can be challenging to manage.
Innovation Solution
A query orchestrator generates a calculation graph by calling multiple engines to analyze the query, adding nodes based on necessary calculations, and executing these nodes in a pre-defined order to process the query, allowing for parallel execution and efficient data blending.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data blending is performed by retrieving data from multiple data sources using different queries, then data from multiple sources can be viewed in a single visualization, but the system processing complexity increases
Solution Approach 1:
The patent segments the query processing into multiple independent engines (e.g., optimization engine, translation engine, execution engine) that each handle specific aspects of query processing. This segmentation allows the system to manage complex multi-source data blending tasks by dividing them into manageable, specialized components rather than attempting to process everything through a single monolithic system.
Solution Approach 2:
The patent introduces an intermediary layer (query optimizer and calculation graph generator) that sits between the user's data blending request and the underlying multiple data sources. This intermediary translates high-level blending queries into optimized execution plans and coordinates the retrieval and combination of data from various sources, shielding the user from the complexity of multi-source data retrieval while managing the complexity internally.
2Productivity
If multiple engines are called to analyze query information and generate calculation graphs, then query processing efficiency is improved, but the device complexity increases
Solution Approach 1:
The patent creates a universal calculation graph framework that can handle multiple types of queries and data sources through a common processing architecture. The same set of engines and graph structures are used regardless of the specific data sources or query types, allowing the system to maintain high productivity across diverse scenarios while avoiding the need for separate specialized systems for each query type.
Solution Approach 2:
The patent performs preliminary actions by pre-defining the calculation graph structure and engine coordination protocols before actual query execution. The system prepares optimization rules, translation templates, and execution plans in advance, so that when queries arrive, they can be rapidly processed through the pre-configured multi-engine architecture without ad-hoc complexity management during execution.
Data Source
AI summary
In one embodiment, a method receives a query for data in a database system and calls a plurality of engines to analyze information for the query. A calculation graph is generated from at least a portion of the plurality of engines where each of the at least a portion of the plurality of engines add a node to the calculation graph based on analyzing of the information for the query. Then, the method executes the query by calling the nodes of the calculation graph. Each node uses metadata added to the node from a respective calculation engine to perform a calculation for the node. Then, a result of the query is output based on the respective calculations performed by the nodes.