Filtering Bitmap for Query Aggregation Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Complex aggregation queries with both grouping and filtering operations in large datasets require significant resources, leading to inefficient processing times due to unnecessary data aggregation and high memory consumption.
Innovation Solution
The creation of a filtering bitmap based on filtering search terms allows for early filtering of query results that do not satisfy the aggregation criteria, reducing the amount of data processed and memory usage by performing filtering aggregation before grouping aggregation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional aggregation query processing is used, then query results are generated, but processing time and resource consumption are excessive
Solution Approach 1:
The patent applies preliminary action by performing filtering aggregation before grouping aggregation. The system first identifies filtering criteria and pre-filters the data set using bitmap indexes, removing irrelevant records before the expensive grouping operation. This early filtering reduces the data volume that needs to be processed in subsequent aggregation steps, directly addressing the excessive query execution time problem.
Solution Approach 2:
The patent segments the aggregation query processing into distinct phases: filtering aggregation phase and grouping aggregation phase. By separating these operations and applying filtering first, the system avoids processing unnecessary data through the entire aggregation pipeline. This segmentation allows independent optimization of each phase and reduces overall processing time.
2Reliability
If complete data aggregation is performed, then accurate query results are obtained, but memory consumption increases significantly
Solution Approach 1:
The patent uses preliminary action with bitmap indexes to identify and mark records that satisfy filtering criteria before aggregation. These bitmaps provide a compact representation of data subsets, allowing the system to filter records early without loading complete aggregation results into memory. This maintains result accuracy while significantly reducing memory consumption during processing.
Solution Approach 2:
The patent extracts and applies filtering conditions separately from the main aggregation process. By extracting the filtering logic and applying it first using bitmap indexes, the system removes irrelevant data before aggregation, keeping only necessary records in memory during the aggregation phase. This extraction approach ensures accurate results while minimizing memory requirements.
3Reliability
If filtering aggregation is performed after grouping aggregation, then correct filtering is applied, but processing efficiency is reduced
Solution Approach 1:
The patent inverts the traditional query processing order by performing filtering aggregation before grouping aggregation, rather than after. This inversion is enabled by bitmap indexes that allow filtering to be applied efficiently on the raw data. The system correctly applies filtering criteria to identify relevant records first, then performs grouping only on the filtered subset, maintaining filtering accuracy while dramatically improving processing efficiency.
Solution Approach 2:
The patent applies preliminary filtering action using bitmap indexes before the grouping operation. The filtering aggregation is performed in advance to identify and retain only records meeting the filtering criteria. This preliminary action ensures filtering accuracy is maintained while improving overall query processing efficiency by reducing the data volume for subsequent grouping operations.
Data Source
AI summary
Embodiments described herein provide a solution for optimizing a generating of query search results. A filtering search term (e.g., a search term that is used in a query to perform filtering aggregations of the query search results) is identified. A filtering bitmap that has a plurality of mapped locations corresponding to data values for the filtering search term is created. As a data value in the filtering search term is encountered during a scan of the query search results, the corresponding mapped location is updated. Each mapped location in the filtering bitmap is read to determine whether the value corresponding to the mapped location satisfies the filtering aggregation. The filtering aggregation can then be performed (e.g., prior to any grouping aggregation) by removing any of the query search results determined, based on the filtering bitmap, as having data values for which the filtering aggregation is not satisfied.