Encoded Data Filtering for Faster Predicate Pushdown Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data query systems face bottlenecks in filtering and scanning encoded data due to the need for data decoding, which compromises query performance and efficiency, especially when using predicate pushdown techniques.
Innovation Solution
A data filtering method and apparatus that determines a target filtering operator and uses a pre-built operator function set tailored to the hardware platform and data encoding features to perform filtering on encoded data before decoding, optimizing the query process by reducing the amount of data that needs to be decoded.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data are compressed and encoded for efficient storage, then storage efficiency is improved, but filtering and scanning performance deteriorates due to the need for decoding
Solution Approach 1:
The patent pre-computes and stores execution functions during the compilation phase that can directly operate on encoded data without requiring full decoding. These execution functions are generated in advance based on the encoding features and data features, enabling filtering operations to be performed on encoded data directly, thus resolving the contradiction between storage efficiency (through encoding) and filtering performance (by avoiding decoding).
2Loss of time
If filtering operators are pushed down to storage layer for preliminary filtering, then query execution time is reduced, but data encoding and decoding complexity increases
Solution Approach 1:
The patent extracts the decoding step from the filtering operation by creating execution functions that work directly on encoded data. Instead of decoding data first and then filtering, the system extracts and applies filtering logic that operates on the encoded representation, thereby reducing query execution time while avoiding the complexity of full decoding and re-encoding processes.
Solution Approach 2:
The patent changes the operational parameters of filtering operators to work with encoded data representations. By generating execution functions that accept encoded data as input and produce filtered encoded data as output, the system modifies the parameter space of filtering operations to accommodate encoded formats, eliminating the need for decoding and simplifying the overall process.
3Device complexity
If general-purpose filtering functions are used for all data types and encoding formats, then system simplicity is maintained, but query performance on specific data features deteriorates
Solution Approach 1:
The patent implements a dynamic function selection mechanism where execution functions are chosen based on the specific data features and encoding features of the target data block. The system queries the pre-built operator function set using these features as indices to select the most appropriate execution function, making the system adaptive to different data types and encoding formats while maintaining simplicity through pre-computation.
Solution Approach 2:
The patent creates a universal execution function set that covers multiple data types and encoding formats through a single query interface. The operator function set acts as a multi-functional repository where different execution functions can be selected based on the input features, providing both simplicity (single query mechanism) and performance (specialized function selection).
Data Source
AI summary
Embodiments of this specification provide data filtering methods, computer-readable media and apparatuses for data queries. The filtering methods can be performed by a target device that serves as a storage node of a database. As an example, a target filtering operator involved in a query is determined. A target data block stored in the target device is obtained. Further, a pre-built operator function set is queried by using a data feature and an encoding feature of the target data block to obtain a target execution function of the target filtering operator. The operator function set is used to record filtering operator execution functions that are suitable for use and that correspond to various data types and various encoding manners. The target data block can be filtered by using the target execution function matched by the query to obtain a filtering result.