Hyperloglog Sketches for Inclusion Coefficient Estimation

The patent changes the fundamental parameter of the sketch data structure from Bottom-k to Hyperloglog. Hyperloglog sketches use probabilistic counting with logarithmic space complexity, allowing accurate estimation even when cardinality exceeds the memory budget. The estimation accuracy is maintained through mathematical properties of hyperloglog sketches that provide constant-factor approximation regardless of the relationship between cardinality and memory size.

The patent makes the sketch data structure dynamic by enabling incremental updates for both insertions and deletions. The hyperloglog sketch maintains the ability to update efficiently: for insertions, it simply adds the new element to the sketch; for deletions, it uses a auxiliary structure (such as a heap or counter) to track and remove elements. This dynamic capability allows the sketch to maintain estimation accuracy over time as data is added or removed from the database.

The patent implements dynamic maintenance with efficient time complexity. For insertions, the operation is O(1). For deletions, the patent uses a auxiliary structure (such as a max-heap or counter array) that allows deletion in O(log k) or O(1) time. This dynamic design ensures that maintenance operations remain computationally efficient even as data changes over time, significantly improving productivity compared to recomputing the entire sketch.