Lazy Materialization in Columnar Scans With Pushed Filters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing query processing in cloud-based columnar databases is inefficient and expensive due to the need to fetch entire columns, especially when filters return no values, leading to throttling and increased run times.
Innovation Solution
Implementing a lazy materialization technique that filters and decodes only relevant columns, using machine learning to determine when to apply this method based on likelihood values exceeding a threshold, thereby reducing unnecessary data retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If entire columns are fetched from cloud-based object storage, then complete data is available for querying, but query cost and processing time increase significantly
Solution Approach 1:
The patent applies preliminary action by pushing filter conditions down to the storage layer before data is fully materialized. The filter expression is evaluated during the data fetching process itself, allowing the system to pre-determine which rows satisfy the query conditions before completing the full column fetch, thereby reducing unnecessary data transfer and processing time.
Solution Approach 2:
The patent segments the data fetching process into multiple stages: first fetching only the filter column to evaluate filter conditions, then conditionally fetching additional columns only for rows that satisfy the filter. This segmentation allows the system to avoid fetching entire columns when not needed, reducing I/O operations and query processing time while maintaining data completeness for relevant rows.
2Reliability
If entire columns are fetched and then filtered, then all possible query results are obtained, but unnecessary data transfer increases cost and processing overhead
Solution Approach 1:
The patent performs preliminary filtering by evaluating filter expressions during the data fetching process rather than after complete materialization. This allows the system to identify and discard non-matching rows early, reducing the amount of data that needs to be transferred and processed subsequently, thereby reducing energy consumption and data transfer costs while maintaining query result accuracy.
Solution Approach 2:
The patent applies partial action by fetching and processing only the portions of columns that are necessary to satisfy the query conditions. Instead of materializing entire columns, the system fetches data in a controlled manner, applying filters incrementally, and only fully materializing columns for rows that ultimately satisfy all filter conditions, thus reducing unnecessary data transfer and processing overhead.
3Loss of energy
If cloud-based storage system throttles expensive queries, then resource usage is controlled, but query run time increases further
Solution Approach 1:
The patent applies preliminary action by pushing filter conditions down to the storage layer, allowing the system to evaluate filter expressions during data fetching. This reduces the amount of data that needs to be transferred and processed by the compute layer, thereby reducing resource consumption and the likelihood of throttling, while maintaining or improving query execution speed through early elimination of non-matching rows.
Data Source
AI summary
Disclosed herein is a method for determining whether to apply a lazy materialization technique to a query run. The method includes receiving a request to perform a new query in a columnar database containing a plurality of columns. A step in the method includes accessing a set of data in a column of the plurality of columns based on the query. The method includes generating an input to a machine-learned model comprising characteristics of the set of data in the column. From the machine-learned model, the method includes generating a likelihood value indicative of whether a filter of a first portion of the set of data in the column has greater efficiency than a download followed by a filter of the set of data in the column. The method further includes comparing the likelihood value to a threshold value. Based on the comparison, the method includes filtering the first portion of the set of data before downloading the set of data if the likelihood value is equal to or above the threshold value.