Scalable Distributed Exact Distinct Counts With Balanced Value Partitions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face scalability issues and inefficiencies in performing exact and multiple exact distinct counts on large datasets due to low distinctness of keys, leading to skewed partitions and excessive memory and processing requirements.
Innovation Solution
Implementing systems and devices that sort datasets by values, partition subsets with the same value together, and generate compound keys for combined fields, allowing for efficient aggregation of exact distinct counts and multiple distinct counts by partitioning on identifiers/values instead of keys.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data is partitioned by keys with low distinctness, then the distinct count can be computed, but partition skew occurs and memory requirements become excessive
Solution Approach 1:
Instead of partitioning by key (which causes skew when keys have low distinctness), the patent inverts the approach by partitioning by identifier/value. This reversal eliminates the partition skew problem while maintaining accurate distinct count computation, as identifiers with high distinctness create more balanced partitions.
Solution Approach 2:
The patent changes the partitioning parameter from key to identifier/value. This parameter change fundamentally alters the partition distribution, transforming skewed partitions (caused by low key distinctness) into balanced partitions (enabled by high identifier distinctness), thereby reducing memory requirements.
2Measurement precision
If multiple distinct counts are performed separately, then each count can be computed, but an extra partition is required increasing complexity
Solution Approach 1:
The patent merges multiple distinct count operations into a single partitioning and processing step. By partitioning once by identifier/value and then performing multiple distinct counts on the same partitioned data, the system eliminates the need for separate partitions for each count operation, reducing overall system complexity.
Solution Approach 2:
The single partition structure serves multiple functions: it enables computation of multiple distinct counts simultaneously while maintaining data organization. This multi-functional partition approach replaces what would traditionally require multiple separate partitions, simplifying the system architecture.
3Ease of operation
If random partitioning is used, then data distribution is simple, but partition skew occurs when distinct values are few
Solution Approach 1:
The patent applies local quality by sorting data by identifier/value before partitioning. This creates locally organized partitions where data with similar identifiers are grouped together, ensuring balanced partition composition. The sorting operation ensures that each partition has a consistent and predictable data distribution pattern.
Data Source
Figure 1~2
Figure 3~5
Figure 4
AI summary
Scalable implementations of exact distinct counts and multiple exact distinct counts in distributed query processing systems are implemented via systems and devices. Distinct counts and multiple exact distinct counts for identifiers/values are performed based on keys. For distinct counts, datasets including data fields are sorted by values of fields and divided into balanced partitions in distributed servers. Subsets of fields with the same value are partitioned together. Key presence is determined for subsets on each partition, and the number of instances for the key are aggregated for exact distinct counts of values. For multiple distinct counts, fields of a dataset are combined by un-pivoting field columns. Compound keys are generated for combined fields from field identifiers of the combined fields and values of another field. Totals of unique values of the combined fields are determined for values in the counted field based on the compound keys.