Virtual Register Cardinality Estimation With RRSE Denoising
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for cardinality estimation in large datasets, such as those used by media monitoring entities, face challenges in accurately determining the number of unique entries or elements due to noise introduced by sharing registers across labels, leading to inaccurate estimates, especially in diverse and large-scale datasets.
Innovation Solution
The implementation of a rank recovery-based spread estimator (RRSE) to correct for noise in virtual HyperLogLog (vHLL) algorithms by using a probabilistic distribution of noise, allowing for accurate cardinality estimation by denoising the virtual register array and applying HyperLogLog (HLL) on the corrected array.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If registers are shared across labels in cardinality estimation, then memory usage is reduced, but measurement precision deteriorates due to noise introduced by register sharing
Solution Approach 1:
The patent extracts and removes the noise component from the cardinality estimation process. By identifying that the noise follows a specific probabilistic distribution (geometric distribution), the system separates the noise from the true cardinality signal and removes it, leaving only the accurate cardinality estimate. This is achieved through the denoising function that takes the noisy estimate and the distribution parameters to produce a corrected estimate.
Solution Approach 2:
The patent implements feedback by using the observed noise characteristics from the shared register array to correct future estimates. The system continuously monitors the noise distribution parameters (mean and variance) and uses this feedback to adjust the cardinality estimation process, improving accuracy over time while maintaining memory efficiency.
2Productivity
If virtual HyperLogLog algorithms are used to estimate cardinality, then processing speed is improved, but measurement precision deteriorates due to noise in the virtual register array
Solution Approach 1:
The patent converts the harmful noise in the virtual register array into a beneficial signal. By recognizing that the noise follows a predictable geometric distribution, the system uses the noise characteristics themselves to correct the estimation. The noise, rather than being a pure obstacle, becomes a source of information about the distribution parameters that helps refine the cardinality estimate.
Solution Approach 2:
The patent introduces an intermediary denoising function that mediates between the noisy virtual HyperLogLog output and the final cardinality estimate. This intermediary layer processes the noisy data through a mathematical transformation that eliminates the noise while preserving the true cardinality signal, allowing the system to maintain both speed and accuracy.
3Device complexity
If global registers are shared across multiple labels, then device complexity is reduced, but reliability deteriorates due to noise interference in cardinality estimation
Solution Approach 1:
The patent changes the parameters of the estimation process by introducing corrections based on the noise distribution parameters (mean and variance). Instead of using the raw noisy estimates, the system transforms these estimates using the observed parameters to produce corrected values. This parameter-based correction maintains system simplicity while improving reliability.
Data Source
AI summary
Methods, apparatus, systems, and articles of manufacture to virtually estimate cardinality with global registers are disclosed. An example apparatus includes processor circuitry to s assign subsets of a sample dataset to a shared global register array, the shared global register array having a first number of registers, the sample dataset selected from a reference dataset of media assets; identify a virtual register array from the shared global register array that includes data elements associated with a label value, the virtual register array including a second number of registers less than the first number of registers; determine a maximum rank value of the label value across the virtual register array; and calculate a cardinality estimate of the label value across the virtual register array based on the second number of registers and the maximum rank value.