Query Execution Grouping for Big Data Store Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database technologies face challenges in efficiently processing multiple queries that require the same portion of data, leading to increased latency and resource constraints due to multiple scans over large data stores, especially in big data environments.
Innovation Solution
A method that groups queries based on metadata to execute them in batches, scanning the data store only once for each group, thereby reducing the number of scans and improving query execution efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple queries requiring the same data are processed separately using existing scanning methodologies, then each query can be executed independently, but the number of scans over the data store increases significantly, leading to increased latency and resource consumption
Solution Approach 1:
The patent combines multiple queries that require the same data into a single scan operation. The query processing system identifies queries with overlapping data requirements and merges them into a unified scan operation, where the data store is scanned only once to satisfy multiple queries simultaneously. This eliminates redundant scans and significantly reduces the time and resources required for query execution.
Solution Approach 2:
The system performs preliminary analysis of incoming queries to identify those that can be grouped together before execution. By pre-processing queries to detect overlapping data requirements and organizing them into groups, the system prepares for efficient batch execution. This preliminary grouping action prevents the need for multiple separate scans and enables optimized processing of query groups.
2Reliability
If the data store is scanned multiple times for different queries, then each query can retrieve its required data, but the disk operations, network I/O operations, and CPU operations increase proportionally
Solution Approach 1:
The patent merges multiple data retrieval operations into a single scan operation. By identifying queries that access the same data blocks and combining them into a unified scan, the system retrieves all required data in one pass through the data store. This single scan operation satisfies multiple queries simultaneously, dramatically reducing disk I/O, network I/O, and CPU operations compared to executing separate scans for each query.
3Adaptability or versatility
If concurrent queries requiring the same data portion are processed using existing methodologies, then each query receives its results, but the same data blocks are read multiple times, causing time-consuming and complex processing
Solution Approach 1:
The patent combines concurrent queries that require the same data into a single processing operation. The system identifies queries with overlapping data requirements and executes them as a group, reading each data block only once regardless of how many queries need it. This merging approach maintains the ability to handle diverse query types and data requirements while dramatically improving processing throughput by eliminating redundant data reads.
Data Source
Figure 1~2
Figure 3~4
Figure 5
AI summary
Embodiment of the present disclosure relates to Big Data. In particular,the present disclosure relates to a method and a server to optimize query execution on a data store. The query execution in the present disclosure is optimized by grouping one or more queries, requiring same portion of data from the data store, into one or more groups. Grouping of the one or more queries into the one or more groups is achieved from one or more metadata included in the one or more queries specified by a user who wishes to retrieve the results based on the one or more metadata. The one or more queries grouped under the one or more groups are executed that involves scanning of the data store only for once. In such way, each query is returned with required results from the data store with minimum latency.