Cloud Data Query AQP Sampling for Low-Latency Refinement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The 'decoupled storage-compute' architecture in cloud-native data warehouses experiences significant data access latency due to network latency between compute and storage nodes, leading to prolonged wait times and compromised user experience.
Innovation Solution
Generate N approximate query processing (AQP) messages with varying sampling parameters, allowing simultaneous data query processing across multiple computing clusters, and send M accurate data query responses to the client, ensuring quick feedback and improved accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If decoupled storage-compute architecture is used, then scalability and elastic scaling capability are improved, but data access latency increases significantly
Solution Approach 1:
The patent segments the data access process into multiple parallel approximate query processing tasks with different sampling parameters. By dividing the single query into N parallel AQP queries, the system can retrieve results from multiple storage nodes simultaneously, reducing the impact of network latency while maintaining the decoupled architecture's scalability benefits.
Solution Approach 2:
The patent merges multiple approximate query processing results with different sampling parameters into a single comprehensive query result. By combining results from N parallel AQP queries, the system achieves both fast response (from lower sampling parameter queries) and high accuracy (from aggregated results), effectively addressing the latency issue while preserving architectural flexibility.
2Speed
If sampling parameter value is decreased, then query response speed is improved, but query result accuracy deteriorates
Solution Approach 1:
The patent dynamically adjusts sampling parameters across multiple parallel queries rather than using a fixed parameter. By submitting N AQP queries with progressively different sampling parameters, the system adapts to balance speed and accuracy requirements, providing fast initial results while continuously refining accuracy through additional queries.
Solution Approach 2:
The patent changes the sampling parameter values across multiple parallel AQP queries. By varying the sampling parameter from low to high values across N queries, the system generates a spectrum of results that balance speed and accuracy, allowing the client to receive quick preliminary results while more accurate results follow sequentially.
3Measurement precision
If multiple AQP messages with different sampling parameters are processed, then query result accuracy is improved, but system complexity increases
Solution Approach 1:
The patent makes the query processing system universal by implementing a standardized AQP message format and processing pipeline that handles multiple sampling parameters uniformly. The same infrastructure processes all N AQP queries, and the result merging mechanism works consistently regardless of the number of parallel queries, reducing the complexity overhead of handling multiple parameters.
Data Source
AI summary
A data query method comprises N AQP messages that are generated based on a data query message. In addition, sampling parameters corresponding to the AQP messages have different values. A data query result may be quickly fed back to a client for an AQP message with a small sampling parameter value. Moreover, a data query result corresponding to each AQP message may be obtained based on the N AQP messages, so that the data query results fed back to the client are increasingly accurate.