Distinct Entity Estimation via Probabilistic Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for determining the number of unique entities in a database system are computationally complex, especially when dealing with records that have multiple field types and incomplete values, as they often require matching and clustering operations.
Innovation Solution
A method that generates match rule keys based on predefined rules and increments counters associated with high-order keys to estimate the number of distinct entities in a single pass over the records, without performing matching or clustering operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If matching and clustering operations are performed to determine unique entities, then measurement precision is improved, but productivity deteriorates due to computational complexity
Solution Approach 1:
The patent replaces the mechanical matching and clustering operations with a probabilistic counting system using hashing functions and counter mechanisms. Instead of performing complex pairwise comparisons to identify duplicates, the system applies hashing functions to transform records into scalar values and uses counter increments/decrements to estimate unique entity counts, dramatically reducing computational complexity while maintaining reasonable accuracy
Solution Approach 2:
The patent changes the parameter space by transforming multi-field records into single scalar hash values. By applying hashing functions that convert complex record structures into comparable scalar parameters, the system enables efficient counting operations without requiring detailed field-by-field matching, thus improving productivity while preserving measurement precision through probabilistic methods
2Productivity
If probabilistic counting with hashing functions is used, then productivity is improved, but device complexity worsens due to handling multiple field types and incomplete values
Solution Approach 1:
The patent implements a universal hashing mechanism that can process multiple field types and handle incomplete values through a single unified approach. The hashing function is designed to work across diverse record structures (relational rows, JSON documents, XML, key-value pairs) without requiring separate processing logic for each type, thereby maintaining productivity while managing device complexity through multi-functionality
Solution Approach 2:
The patent introduces hashing functions as intermediary components that mediate between diverse record structures and the counting mechanism. These hashing functions serve as a buffer layer that transforms various input formats into a standardized scalar representation, simplifying the overall system architecture and reducing the complexity of handling multiple field types and incomplete values directly in the counting logic
3Measurement precision
If pair-wise comparison of all records is performed, then measurement precision is improved, but loss of time worsens due to computational complexity
Solution Approach 1:
The patent substitutes the time-consuming pair-wise comparison mechanism with a probabilistic hashing and counting system. Instead of comparing every record against every other record to identify duplicates, the system uses hashing functions to map records to scalar values and employs counter mechanisms to estimate unique entity counts, dramatically reducing processing time while maintaining acceptable measurement precision through probabilistic methods
Data Source
AI summary
A method and system for estimating a number of distinct entities in a set of records are described. For each one of a subset of records, a set of match rule keys are generated based on a set of match rules. Each match rule from the set of match rules defines a match between records, and each match rule key from the set of match rule keys includes at least a key field value. A high order key for the record is determined based on the match rule keys, and a counter associated with the high order key is incremented. When each record from the subset of records has been processed by determining the match rule keys, and incrementing the counter(s) of the high order keys, a sum of a number of counters that have a non-zero value is performed to estimate the distinct entities in the records.