Frequency Distribution Reporting for Privacy-Safe Impression Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for monitoring Internet media access, such as server logs, are susceptible to over-counting and under-counting due to caching and tampering, and deduplication techniques require significant storage and computation resources while compromising user privacy.
Innovation Solution
Implementing an augmented HyperLogLog (HLL) protocol to dynamically deduplicate audience impressions across multiple database proprietors using non-personally identifiable information (PII) data, allowing for on-the-fly deduplication with reduced storage and computation requirements, and enhancing demographic coverage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If server logs are used to monitor Internet media access, then access monitoring is achieved, but over-counting and under-counting occur due to caching and tampering
Solution Approach 1:
The patent introduces an intermediary deduplication system that sits between data collection and analysis. This system uses probabilistic data structures (like Bloom filters) and heuristics to identify and remove duplicate impressions across multiple database proprietors, thereby improving both reliability and measurement precision without requiring direct access to raw user data
Solution Approach 2:
The patent creates simplified copies or representations of user interaction data through hashing and aggregation techniques. Instead of working with complete user profiles and raw interaction logs, the system uses hashed identifiers and aggregated metrics that preserve deduplication functionality while reducing vulnerability to tampering and caching issues
2Measurement precision
If deduplication techniques are implemented to remove duplicate impressions, then counting accuracy is improved, but storage and computation resources increase significantly
Solution Approach 1:
The patent changes the parameters of data representation by using hashed identifiers instead of raw user data, and by implementing progressive aggregation where data is summarized at multiple levels of granularity. This allows the system to maintain deduplication precision while reducing storage requirements by orders of magnitude
Solution Approach 2:
The patent implements partial deduplication by focusing on the most common duplication patterns and applying deduplication heuristics selectively rather than exhaustively. This approach achieves sufficient precision for practical purposes while avoiding the computational overhead of complete deduplication across all possible data combinations
3Measurement precision
If traditional deduplication methods are used to identify unique users, then duplicate impressions are removed, but user privacy is compromised
Solution Approach 1:
The patent extracts only the essential information needed for deduplication (hashed user identifiers) while leaving out all personally identifiable information. This extraction approach maintains the ability to identify and count unique users accurately while preserving user privacy by never collecting or storing sensitive personal data
Solution Approach 2:
The patent uses disposable, ephemeral data representations for user identification that are created, used for deduplication, and then discarded. These temporary hashed identifiers serve their purpose of unique user identification without being stored long-term or used for any purpose beyond deduplication, thereby minimizing privacy risks
Data Source
AI summary
Disclosed examples access data from a database, the data stored across multiple registers of the database; determine (a) a maximum rank for each of the multiple registers and (b) a maximum rank count for each of the multiple registers; determine a frequency distribution based on the maximum ranks and the maximum rank counts; and generate a report including at least one of the frequency distribution, the maximum ranks, or the maximum rank counts.