Horizon Histogram Bucket Allocation for Memory-Constrained Frequency Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data analysis methods for identifying items that occur above a certain frequency, such as iceberg queries, are memory-intensive and inefficient, especially when dealing with large datasets and numerous distinct values.
Innovation Solution
The approach allocates a limited number of count storage buckets based on a threshold frequency, allowing for parallel evaluation of data subsets, merging candidate sets, and determining actual high-frequency values by comparing frequencies with the threshold.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If memory buckets are allocated to count each occurrence of each distinct value, then measurement precision of frequency is improved, but device complexity (memory usage) worsens
Solution Approach 1:
The patent segments the data set into multiple partitions and processes each partition separately with its own limited set of count storage buckets. This segmentation allows the system to handle large data sets with many distinct values using limited memory by processing data in manageable chunks rather than requiring all memory buckets to be allocated simultaneously for all distinct values across the entire data set.
Solution Approach 2:
The patent introduces a new dimension of processing by dividing the data set into multiple partitions and processing them in parallel. This dimensional approach transforms the single-pass sequential processing model into a multi-pass parallel model, where each partition is processed independently with limited memory resources, thereby reducing the overall memory requirement while maintaining frequency measurement accuracy.
2Device complexity
If the entire data set is analyzed serially, then device complexity is reduced, but productivity worsens
Solution Approach 1:
The patent segments the data set into multiple partitions that can be processed in parallel. Each partition is analyzed separately using the same frequency counting algorithm, but the parallel execution of multiple partitions significantly improves processing throughput compared to serial analysis of the entire data set, while maintaining the simplicity of the underlying counting mechanism.
Solution Approach 2:
The patent performs preliminary actions by dividing the data set into partitions and pre-processing each partition to identify candidate high-frequency values before merging results. This preliminary processing of individual partitions enables subsequent parallel aggregation of results, improving overall productivity while keeping each individual processing step relatively simple.
3Device complexity
If count storage buckets are limited, then device complexity is reduced, but measurement precision worsens
Solution Approach 1:
The patent segments the counting process into multiple passes over different data partitions. Each pass uses a limited set of count storage buckets, but by processing multiple partitions sequentially or in parallel across multiple passes, the system achieves accurate frequency measurements for all distinct values in the entire data set without requiring all buckets to be allocated simultaneously.
Solution Approach 2:
The patent employs periodic action by repeatedly applying the frequency counting algorithm to different partitions of the data set. Each pass through a partition periodically updates the count storage buckets, and this periodic processing across multiple partitions accumulates accurate frequency information over time, compensating for the limited size of the buckets available at any single moment.
Data Source
AI summary
Values that occur above a threshold frequency for certain characteristic(s) of a data set are identified. A limited number of count buckets are allocated based on the threshold. Buckets store proxy counts for identifying candidate sets of values rather than actual counts. The data set is divided and each portion is analyzed separately, by iterating through each item in that portion. During each iteration, depending on an item's value(s), a bucket is incremented, all buckets are decremented, or a bucket is assigned or reassigned to count different value(s). A candidate set of values and associated counts is selected for a portion based on the buckets. The candidate sets for each portion are merged and, in some embodiments, filtered based on the associated counts. Actual frequencies are then determined for the values that remain in the merged candidate set.