Ordered Statistics Cardinality Estimation for Large Duplicate Datasets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for determining the number of unique entries in large datasets, such as media asset databases, are computationally infeasible and prone to errors due to the growth of duplicate entries and the complexity of large databases.
Innovation Solution
The use of ordered statistics, specifically through the MaxSketch and MinSketch estimators, to estimate cardinality by determining the maximum and minimum order statistics in a sample dataset, allowing for efficient estimation of unique entries and intersection cardinality between datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If exact counting methods are used to determine unique entries in large datasets, then measurement precision is improved, but computational complexity and time consumption increase significantly
Solution Approach 1:
The patent segments the large dataset into multiple smaller subsets and processes each subset independently to determine order statistics. This segmentation allows parallel processing and reduces the computational burden compared to processing the entire dataset as a single unit, thereby reducing computation time while maintaining estimation accuracy.
Solution Approach 2:
The patent uses disposable hash functions that are computationally inexpensive and can be applied rapidly to each data element. These hash functions transform data elements into values suitable for order statistic calculation without requiring complex or time-consuming computational resources, enabling fast cardinality estimation.
2Measurement precision
If comprehensive data processing is performed on large datasets, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent extracts only the essential information needed for cardinality estimation by computing order statistics (minimum, maximum, or other ranked values) from hash function outputs. Instead of processing and storing comprehensive data about all elements, the system extracts and retains only the critical order statistic values, significantly reducing system complexity while maintaining estimation precision.
Solution Approach 2:
The patent changes the parameter representation by transforming raw data elements through hash functions into a different parameter space where order statistics can be efficiently computed. This parameter transformation simplifies the processing requirements and reduces the complexity of the processing system while preserving the ability to accurately estimate cardinality.
3Measurement precision
If traditional counting methods are used in media tracking, then measurement precision is improved, but productivity decreases
Solution Approach 1:
The patent applies preliminary hashing to data elements before they are fully processed or stored. By computing hash values and determining order statistics in advance during data ingestion, the system prepares the data for rapid cardinality estimation without requiring complex post-processing operations, thereby increasing processing throughput while maintaining tracking accuracy.
Solution Approach 2:
The patent replaces traditional mechanical counting mechanisms with statistical estimation based on order statistics of hash function outputs. This substitution eliminates the need for maintaining and processing complex counting structures, enabling faster data processing throughput while achieving comparable or superior measurement precision in media tracking applications.
Data Source
AI summary
Methods, apparatus, systems, and articles of manufacture to estimate cardinality through ordered statistics are disclosed. In an example, an apparatus includes processor circuitry to selects a sample dataset from a first reference dataset of media assets and partitions the sample dataset into m mutually exclusive subsets of approximately equal size. The processor circuitry then estimates a ratio of a sample weighted average and empirical cumulative distribution of an approximately largest order statistic from at least one of the m subsets and generates an estimate of a total cardinality of the first reference dataset by multiplying the ratio by approximately m.