Distributed Query Distinct Count Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing solutions for exact distinct counts in large datasets, such as big data, face scalability issues due to low distinctness of keys, leading to memory and processing inefficiencies and skewed partitions, especially when performing multiple distinct counts.
Innovation Solution
The approach involves sorting datasets by values, dividing them into balanced partitions, determining key presence, and aggregating instances for exact counts, while for multiple counts, data is un-pivoted and combined using compound keys to simplify processing and reduce resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data is partitioned by keys for distinct counts, then distinct count accuracy is improved, but partition balance deteriorates when key distinctness is low
Solution Approach 1:
The patent segments the distinct count problem into two phases: first partition by key to ensure all values for each key are grouped together (maintaining count accuracy), then further segment by value within each partition to enable balanced aggregation. This two-level segmentation resolves the contradiction by maintaining key-based grouping while achieving value-based balance.
Solution Approach 2:
The patent introduces a second dimension of partitioning by switching from key-based partitioning to value-based partitioning. This dimensional switch allows the system to maintain the benefits of key-based grouping while achieving the balance benefits of value-based distribution, effectively resolving the contradiction through dimensional transformation.
2Measurement precision
If multiple distinct counts are performed separately, then count accuracy is improved, but processing time increases
Solution Approach 1:
The patent merges multiple separate distinct count operations into a single unified processing pipeline. By performing all distinct counts in one pass through the data with a single partitioning and aggregation phase, the system maintains accurate results while eliminating the time overhead of multiple separate executions.
Solution Approach 2:
The patent performs preliminary grouping by key and sorting by value before the counting phase, so that when multiple distinct counts are needed, the data is already organized optimally. This preliminary organization enables efficient simultaneous counting of multiple values without reprocessing the raw data.
3Reliability
If separate partitions are created for each count operation, then count independence is improved, but memory usage increases
Solution Approach 1:
The patent creates a universal partitioning structure that serves multiple counting operations simultaneously. A single partitioning by key followed by sorting by value enables independent counting of any number of different values without creating separate partitions for each, making the partitioning system multi-functional and memory-efficient.
Solution Approach 2:
The patent discards the intermediate key-based partitions after initial grouping, recovering memory space while maintaining the grouped data structure needed for subsequent value-based counting. This allows multiple distinct counts to be performed on the same organized data without retaining redundant partition structures in memory.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
Scalable implementations of exact distinct counts and multiple exact distinct counts in distributed query processing systems are implemented via systems and devices. Distinct counts and multiple exact distinct counts for identifiers/values are performed based on keys. For distinct counts, datasets including data fields are sorted by values of fields and divided into balanced partitions in distributed servers. Subsets of fields with the same value are partitioned together. Key presence is determined for subsets on each partition, and the number of instances for the key are aggregated for exact distinct counts of values. For multiple distinct counts, fields of a dataset are combined by un-pivoting field columns. Compound keys are generated for combined fields from field identifiers of the combined fields and values of another field. Totals of unique values of the combined fields are determined for values in the counted field based on the compound keys.