Bitmap Filtering for Funnel Query Execution Speed
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems for executing funnel queries are inefficient, particularly in e-commerce platforms, as they require traversing all sequences to determine which include specific event sequences, leading to slow query execution times.
Innovation Solution
The implementation of a bitmap system to eliminate unnecessary sequences during funnel queries, combined with storing sequences across multiple data partitions covering short time periods, and using memory to track user events across partitions, allows for faster query execution by reducing the number of sequences to traverse and optimizing data partition coverage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all sequences are traversed to determine which include specific event sequences, then complete accuracy is achieved, but query execution time increases significantly
Solution Approach 1:
The patent creates bitmaps in advance that mark which sequences contain which events. During query execution, these pre-computed bitmaps are used to quickly identify candidate sequences without traversing all sequences from scratch, thus maintaining accuracy while reducing execution time.
Solution Approach 2:
The patent extracts only the relevant sequences for filtering based on the query criteria. By using bitmaps to identify and extract only those sequences that contain the required events, the system avoids unnecessary traversal of sequences that cannot possibly match the query, thereby reducing execution time while preserving accuracy.
2Device complexity
If data is stored in a single large partition, then data structure simplicity is maintained, but query performance degrades for long time periods
Solution Approach 1:
The patent divides the data storage into multiple partitions, each covering a specific time period. This segmentation allows queries to be executed by only traversing relevant partitions rather than the entire dataset, significantly improving query performance for long time periods while maintaining manageable partition sizes.
Solution Approach 2:
The patent introduces a time-dimension to the data partitioning strategy. By organizing partitions along the time axis, the system can efficiently handle queries spanning long periods by selectively accessing only the necessary time-range partitions, thus improving productivity without excessive complexity.
3Productivity
If multiple data partitions are used to cover different time periods, then query speed for short periods is improved, but system complexity increases
Solution Approach 1:
The patent segments data into multiple time-based partitions, allowing fast queries by only accessing relevant partitions. This segmentation improves query speed for short periods while keeping each partition manageable in size and scope.
Solution Approach 2:
The patent introduces bitmaps as an intermediary data structure that bridges the gap between multiple partitions and the query processing logic. These bitmaps provide a standardized interface for filtering sequences across partitions, reducing the perceived complexity of managing multiple partitions while maintaining the performance benefits.
4Measurement precision
If memory is used to track user events across partitions, then funnel query accuracy is maintained, but memory usage increases
Solution Approach 1:
The patent uses bitmaps as a compact copy of sequence information that can be efficiently stored in memory. Instead of storing complete sequence data, only essential event presence information is copied into bitmap structures, maintaining query accuracy while using minimal memory resources.
Solution Approach 2:
The patent changes the representation of user event data from detailed sequence storage to compact bitmap encoding. This parameter change transforms the data from a verbose format to a space-efficient format, enabling accurate funnel queries across partitions while keeping memory usage low through bitwise operations.
Data Source
AI summary
Data may be queried and analyzed in order to draw insights. One type of data query that may be performed is a funnel query. A funnel query is a query characterized by a sequence of events, e.g.: “In the last N days, how many unique users performed event A, then event B, and then event C”. Systems and methods for performing funnel queries are provided herein. In some embodiments, the speed at which a computer can answer a funnel query may be increased. In some embodiments, a bitmap is used to eliminate one or more sequences of events that would otherwise need to be traversed during the funnel query. In some embodiments, a sequence of events is stored across multiple data partitions, each data partition covering a different period of time.