Database Frequency Distribution from HyperLogLog Register Ranks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for monitoring Internet media access, such as server logs, are susceptible to over-counting and under-counting due to caching and tampering, and deduplication techniques require massive data storage and privacy concerns, making it difficult to accurately determine unique audience reach across multiple platforms.
Innovation Solution
Implementing a HyperLogLog (HLL) deduplication technique that uses a modified HLL protocol to dynamically deduplicate audience data across multiple database proprietors, leveraging non-personally identifiable information (PII) to estimate unique audience counts with reduced storage and computation, and applying Bayesian analysis to refine frequency distributions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If server logs are used to monitor Internet media access, then access data can be collected, but over-counting and under-counting occur due to caching and tampering
Solution Approach 1:
The patent introduces an intermediary deduplication system that processes access data from multiple server logs before final counting. This intermediary layer applies probabilistic data structures (hyperloglog) to identify and eliminate duplicate impressions caused by caching and tampering, thereby improving both measurement precision and reliability without requiring direct trust in individual server logs
Solution Approach 2:
The patent replaces traditional mechanical deduplication methods (which require storing and comparing all raw access records) with a probabilistic algorithmic approach using hyperloglog data structures. This substitution reduces the computational and storage overhead while maintaining accurate deduplication, addressing the reliability issues of server logs without proportionally increasing system complexity
2Measurement precision
If deduplication techniques are applied to eliminate duplicate impressions, then accurate unique audience counts can be achieved, but massive data storage is required
Solution Approach 1:
The patent fundamentally changes the parameter of data representation by using probabilistic data structures (hyperloglog) instead of storing complete access records. This parameter change allows the system to maintain accurate unique audience counts while reducing storage requirements from gigabytes to kilobytes, as the hyperloglog structure only needs to store compact hash-based summaries rather than full impression data
Solution Approach 2:
The patent extracts only the essential information needed for deduplication (hashes of access identifiers) while discarding redundant data (full access records, timestamps, user details). This extraction approach enables accurate unique counting without storing massive amounts of raw data, directly resolving the contradiction between precision and storage quantity
3Measurement precision
If deduplication techniques process all access data, then unique audience reach can be determined, but privacy concerns arise from handling personal information
Solution Approach 1:
The patent extracts and processes only non-personally identifiable information (hashes of device identifiers or IP addresses) while deliberately excluding personal information from the deduplication process. This extraction strategy enables accurate unique audience measurement without exposing or storing sensitive user data, thereby maintaining precision while eliminating privacy harms
Solution Approach 2:
The patent uses temporary, disposable hash values for deduplication that are generated, processed, and immediately discarded. These cheap computational objects serve their purpose in identifying unique audiences without creating persistent records of personal information, thus achieving measurement precision without long-term privacy risks
4Quantity of substance
If HyperLogLog protocol is used to estimate cardinality, then storage needs are reduced, but frequency distribution information is lost
Solution Approach 1:
The patent segments the hyperloglog data structure into multiple independent registers or buckets, each maintaining its own cardinality estimation. This segmentation allows the system to preserve frequency distribution information by analyzing the distribution of maximum values across different segments, thereby recovering frequency data that would otherwise be lost in a monolithic hyperloglog structure while maintaining reduced storage requirements
Data Source
AI summary
Disclosed examples access data from a database, the data stored across multiple registers of the database; determine (a) a maximum rank for each of the multiple registers and (b) a maximum rank count for each of the multiple registers; determine a frequency distribution based on the maximum ranks and the maximum rank counts; and generate a report including at least one of the frequency distribution, the maximum ranks, or the maximum rank counts.