Data Virtualization Query Pushdown for Latency-Aware Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data virtualization systems face performance bottlenecks due to inefficient query operation delegation, network latency, and inconsistent data semantics, leading to slow query response times and suboptimal query result generation.
Innovation Solution
A processor set determines the capabilities of data sources to split query functions and operations for remote and local execution, optimizing query execution by delegating supported operations remotely and handling unsupported operations locally, while normalizing results to ensure consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If query operations are executed remotely on data sources, then network data transfer is minimized and query execution efficiency is improved, but query response time increases due to network latency
Solution Approach 1:
The patent segments query operations into two categories: push-down operations that can be executed remotely on data sources, and non-push-down operations that must be executed locally at the data virtualization system. This segmentation allows the system to minimize network data transfer by executing as much as possible remotely while maintaining query response time by handling unsupported operations locally without requiring full result set retrieval.
Solution Approach 2:
The patent applies local quality by determining capabilities of individual data sources to identify which operations can be pushed down to remote execution. Each data source is evaluated for its specific capabilities, and query operations are delegated only to data sources that support them, optimizing execution efficiency while accounting for network latency constraints.
2Productivity
If query operations are delegated to remote data sources, then query execution performance is improved, but data semantics consistency becomes inconsistent across different data sources
Solution Approach 1:
The patent introduces an intermediary layer (the data virtualization system) that maintains a unified schema and semantics across multiple heterogeneous data sources. This intermediary translates and standardizes data from different sources, ensuring semantics consistency while allowing remote execution to improve performance. The system acts as a mediator between the query and diverse data sources, harmonizing their different data models.
3Stability of the object's composition
If all query operations are executed locally at the data virtualization system, then data semantics consistency is maintained, but network data transfer increases and query execution efficiency decreases
Solution Approach 1:
The patent segments query processing into push-down operations (executed remotely to improve efficiency) and non-push-down operations (executed locally to maintain semantics consistency). This segmentation eliminates the need to retrieve entire result sets to the data virtualization system, reducing network data transfer while maintaining data semantics consistency through the intermediary layer.
4Ease of operation
If the system retrieves entire result sets to the data virtualization system, then complete query processing is enabled, but network data transfer increases and query response time increases
Solution Approach 1:
The patent extracts only the necessary operations (push-down operations) that can be executed remotely on data sources, leaving non-push-down operations to be executed locally. This extraction eliminates the need to retrieve entire result sets across the network, significantly reducing network data transfer and query response time while maintaining complete query processing capability through the combination of remote and local execution.
Data Source
AI summary
A computer-implemented method for optimizing query execution in a data virtualization system is provided. A processor set receives a query using the data virtualization system. The processor set determines capabilities for the data source to identify a portion of functions and a portion of query operations from the query to be delegated for remote execution on the data source. The processor set splits functions and query operations in the query for remote execution and local execution based on the portion of functions and the portion of query operations from the query. The processor set receives results from the data source from remote execution of the query using the data virtualization system. The processor set merges the results from the data source with second result from local execution of the query on the data virtualization system to generate a merged result. The processor set returns the merged result to the application.