Presence Bitmaps for Count-Distinct Query Optimization

The patent segments the large data set into multiple partitions based on dimension values (e.g., partitioning telephone numbers by area code). Each partition is processed separately to identify unique keys, and results are aggregated. This segmentation reduces the computational burden on any single processing unit and enables parallel execution, significantly reducing query execution time while maintaining accuracy.

The patent performs preliminary processing by pre-computing and storing presence bitmaps that indicate the presence of unique keys in different partitions. These bitmaps are prepared in advance and can be quickly queried without re-processing the entire data set. This preliminary action transforms the expensive count-distinct operation into a simple bitmap query, dramatically reducing execution time.

The patent creates compact bitmap representations (copies) of the presence information for unique keys in each partition. Instead of processing the full multi-dimensional data set during query execution, the system uses these pre-generated bitmap copies that occupy minimal memory. This copying approach maintains complete analysis capability while reducing memory usage and computational resources by several orders of magnitude.

The patent transforms the data representation from storing actual multi-dimensional records to storing compact bitmap parameters that encode presence information. This parameter change from detailed records to compressed bitmaps dramatically reduces the memory footprint and processing requirements while preserving the ability to accurately determine unique key counts across partitions.