Remote Storage Grouping to Speed Distinct-Value Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Caching data pages at the query engine provides minimal performance benefits for analytical queries with large workloads, leading to performance penalties due to cache pollution and reduced availability of relevant data for subsequent queries.
Innovation Solution
Shift some query processing to the storage level, leveraging parallel performance capabilities of a distributed storage system, and perform operations like requests for distinct items efficiently at the storage level to reduce data transfer and enhance query speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data pages are cached at the query engine for quick access, then individual query speed is improved, but analytical queries with large workloads cause cache pollution and reduce performance of subsequent queries
Solution Approach 1:
The patent segments query processing into two parts: filtering operations are performed at the storage service level on data pages before transfer, while the query engine processes only the filtered results. This segmentation prevents large analytical query results from polluting the cache, as only small filtered result sets are cached at the query engine, resolving the contradiction between fast individual query access and sustained productivity for subsequent queries
Solution Approach 2:
The storage service performs filtering operations preliminarily on data pages before transferring them to the query engine. By pre-filtering data at the storage level, the system reduces the volume of data that needs to be cached and processed at the query engine, thereby preventing cache pollution from large analytical queries while maintaining fast access for subsequent queries
2Reliability
If large analytical queries are processed at the query engine, then query completeness is improved, but network bandwidth is overwhelmed and performance degrades
Solution Approach 1:
The patent extracts the filtering operation from the query engine and moves it to the storage service level. The storage service filters data pages before transfer, extracting only the necessary filtered results that the query engine needs to process. This reduces network bandwidth utilization by transmitting only essential data, while the query engine maintains query completeness by receiving pre-filtered results
Solution Approach 2:
Filtering operations are performed preliminarily at the storage service before data is transferred to the query engine. This preliminary filtering reduces the volume of data transmitted over the network, decreasing network bandwidth utilization, while ensuring that the query engine receives complete and accurate filtered results for reliable query processing
3Measurement precision
If all query processing is performed at the query engine, then query accuracy is improved, but query speed decreases due to data transfer overhead
Solution Approach 1:
The patent segments query processing between the storage service and query engine: the storage service performs filtering operations on data pages, while the query engine processes the filtered results. This segmentation enables parallel processing and reduces data transfer overhead, improving query speed while maintaining query accuracy through coordinated processing at both levels
Solution Approach 2:
The storage service performs filtering operations preliminarily on data pages before transferring them to the query engine. This preliminary processing reduces the amount of data that needs to be transferred and processed at the query engine, thereby improving query speed. The query engine then processes the pre-filtered results to ensure query accuracy is maintained
Data Source
AI summary
Remote grouping operations may be used to perform queries for distinct values. A query may be received at a query engine that includes a request for distinct items from a data set stored in a data store. To perform the query, a request may be made to the data store to group items from the data set by value when returned from the data store to the query engine. The query engine may then evaluate the items grouped by value that are returned from the data store to determine distinct items as part of performing the query.