In-Memory Predicate Evaluation for Columnar Data Scans
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current techniques for evaluating query predicates during an in-memory scan of columnar data require row stitching and decompression, leading to unnecessary overhead, especially when evaluating join predicates across multiple tables, which incurs significant computational and communication costs.
Innovation Solution
The approach involves pushing down qualifying query predicates to be evaluated directly against in-memory columnar units (IMCUs) during the scan, generating implied predicates for complex conditions that cannot be directly evaluated, and applying join filters to minimize data redistribution in a distributed database cluster.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If row stitching and decompression are performed during in-memory scan, then predicate evaluation can be performed on row-major data, but computational overhead increases significantly
Solution Approach 1:
Instead of converting columnar data to row-major format for predicate evaluation, the patent inverts the approach by evaluating predicates directly on columnar data in its native format. This eliminates the need for row stitching and decompression operations, resolving the contradiction between maintaining predicate evaluation capability and reducing computational overhead.
Solution Approach 2:
The patent changes the fundamental parameter of data representation from row-major to columnar format for predicate evaluation. By modifying how data is organized and accessed during scan operations, the system achieves efficient predicate evaluation without requiring format conversion, thus reducing computational overhead while maintaining evaluation capability.
2Reliability
If join predicates are evaluated on all row combinations from multiple tables, then complete join results are obtained, but computational and communication costs increase significantly
Solution Approach 1:
The patent applies preliminary filtering actions by evaluating predicates directly on columnar data during the scan phase, before join operations are performed. This preliminary evaluation filters out non-matching rows early in the processing pipeline, reducing the number of row combinations that need to be evaluated during the join operation, thus maintaining result completeness while reducing computational and communication costs.
Solution Approach 2:
The patent segments the join evaluation process into distinct phases: (1) columnar scan with direct predicate evaluation to filter rows, and (2) join operation on filtered results. This segmentation allows predicate evaluation to occur independently on each table's columnar data before the join, reducing the data volume that needs to be communicated and processed during the actual join operation.
3Adaptability or versatility
If data is broadcast from one node to multiple target nodes in a distributed cluster, then join operations can be performed on distributed data, but inter-process communication overhead increases
Solution Approach 1:
The patent performs preliminary predicate evaluation on columnar data at the source node before data broadcast. This preliminary filtering reduces the volume of data that needs to be broadcast to target nodes, maintaining distributed join capability while minimizing inter-process communication time by sending only relevant filtered results rather than complete data sets.
Data Source
AI summary
Techniques are described herein for filtering data from a table during an in-memory scan. Predicates are pushed to in-memory scan to avoid scanning unnecessary columnar units and reduce the overhead of decompressing, row stitching and distributing data during evaluation. Techniques are described herein for generating implied predicates that have conditions on single columns from complex predicates that have multiple conditions on the same column, which can be evaluated during an in-memory scan. Techniques are also described herein to reduce the overhead of a table scan for processing a join query. When redistributing a first table for performing a hash-join, the nodes performing an in-memory scan of the first table may create a filter that tracks unique values from the join key. Data from the second table is only processed and transferred to other nodes in the cluster if the values from the join key pass through the filter.