Grouping Vector Hash Join for OLAP Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional OLAP query processing methods incur high invalid data transmission costs, repeated hash computation costs, and increased processing delays due to sequential operations and heterogeneous storage and computing platforms, especially when dealing with multiple table joins and large data volumes.
Innovation Solution
A hash multi-table joining method based on a grouping vector, which divides the joining and aggregation computation into two independent stages using a vector index, allowing for asynchronous pipeline processing and optimizing the computation workload distribution across different platforms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If sequential joining operations are performed according to query tree path, then joining correctness is ensured, but invalid data transmission cost increases significantly
Solution Approach 1:
The patent segments the query processing into two independent stages: joining stage and aggregation stage. The joining stage generates intermediate results with grouping attributes, while the aggregation stage performs hash grouping aggregation. This segmentation allows invalid data to be filtered out earlier in the process, reducing unnecessary data transmission while maintaining joining correctness through the structured two-stage approach.
Solution Approach 2:
The patent performs preliminary grouping operations during the joining stage, creating intermediate results that are organized by grouping attributes before the aggregation stage. This preliminary action prepares the data in advance, allowing the aggregation stage to process only relevant data without transmitting invalid records, thus reducing data transmission cost while ensuring correctness.
2Measurement precision
If traditional hash grouping aggregation computation is performed on joining results, then aggregation accuracy is ensured, but repeated hash computation cost increases
Solution Approach 1:
The patent extracts the grouping attribute computation from the traditional hash grouping aggregation process and performs it separately during the joining stage. By taking out the grouping attribute generation as a distinct preliminary operation, the system avoids repeated hash computations during aggregation while ensuring aggregation accuracy through the structured two-stage approach with dedicated aggregation operations.
3Reliability
If data is accessed from three different data sets on heterogeneous platforms, then query completeness is achieved, but processing delay increases
Solution Approach 1:
The patent segments the query processing workload into joining operations and aggregation operations that can be distributed across different heterogeneous platforms. The joining stage can utilize platforms optimized for join operations, while the aggregation stage uses platforms suited for aggregation computations. This segmentation maintains query completeness by processing all necessary data sets while reducing processing delay through optimized platform utilization and parallel execution of independent stages.
4Reliability
If GROUP-BY attributes are transmitted between query trees, then grouping correctness is maintained, but data transmission cost increases
Solution Approach 1:
The patent performs preliminary grouping attribute generation during the joining stage, creating intermediate results that are already organized by grouping attributes. This preliminary action eliminates the need to transmit GROUP-BY attributes between query trees separately, as they are already embedded in the intermediate results. Grouping correctness is maintained through this structured approach while data transmission cost is reduced by avoiding redundant attribute transmission.
Data Source
AI summary
The disclosure includes aspects of a system and/or method including a hash multi-table joining implementation method based on a grouping vector, which includes the following steps: 1) rewriting an SQL query command, and dividing a complete OLAP query command into a subtask of selecting-projecting-grouping-joining operation and an subtask of aggregating operation; 2) creating and generating grouping vector metadata corresponding to a GROUP-BY statement in an SQL command through the subtask of selecting-projecting-grouping-joining operation, and creating a vector index as an output result of the subtask of selecting-projecting-grouping-joining operation; 3) executing aggregation computation based on the vector index through the subtask of aggregating operation, and storing an aggregation computation result in a corresponding unit of a grouping vector aggregator with the same length as the grouping vector; and 4) merging the aggregation computation result in the grouping vector aggregator with the grouping vector metadata created in the joining operation subtask, and outputting the merged data as a result set of the SQL query command. The disclosure can be widely applied to the field of OLAP data query.