Streaming Data Set Intersection Using Bottom-k Sketches
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional databases face challenges in performing set operation queries on streaming data due to issues with data freshness, missing records, synchronization, and high storage and processing requirements, making it difficult to efficiently answer set intersection queries, especially when dealing with dynamic numerical thresholds and large datasets.
Innovation Solution
The use of bottom-k sketches to generate and update data structures that allow for efficient estimation of set intersection cardinality, enabling quick and accurate responses to queries by partitioning data into quantiles and maintaining sketches without the need for repeated passes through the entire dataset, thus reducing storage and processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional databases are used to store and query streaming data, then data can be stored and retrieved, but set operation queries face problems with data freshness, missing records, synchronization, and require great storage space and long processing time
Solution Approach 1:
The patent divides the streaming data into multiple quantiles (e.g., bottom 25%, next 25%, etc.) and maintains separate bottom-k sketches for each quantile. This segmentation allows the system to process queries on smaller subsets of data rather than the entire dataset, significantly reducing processing time for set operations while maintaining data freshness through continuous updates of individual quantile sketches.
Solution Approach 2:
The patent extracts only the necessary information (bottom-k elements) from the streaming data for each quantile and stores it in compact sketches, rather than storing the complete streaming data. This extraction reduces storage requirements and enables faster query processing by working with the extracted summaries rather than the full datasets.
2Productivity
If traditional databases store streaming data for set operations, then data can be queried, but storage space requirements are great and update operations are slow
Solution Approach 1:
The patent extracts only the bottom-k elements from each quantile of streaming data and stores them in compact sketches, rather than storing the complete streaming data. This extraction reduces storage requirements significantly while enabling frequent updates, as only the sketch data needs to be updated rather than the entire dataset.
Solution Approach 2:
The patent changes the storage parameter from storing complete streaming data to storing compressed sketches of bottom-k elements. This parameter change enables the system to maintain constant sketch size regardless of data volume, allowing for frequent updates without proportional increases in storage space or update time.
3Reliability
If numerical thresholds are updated frequently in streaming data, then data freshness is maintained, but processing time and computational resources increase
Solution Approach 1:
The patent segments the data into quantiles and maintains separate bottom-k sketches for each quantile. When numerical thresholds need to be updated, only the affected quantile sketches need to be regenerated and updated, rather than processing the entire dataset. This segmentation enables frequent threshold updates while maintaining data freshness without proportional increases in processing time.
Solution Approach 2:
The patent pre-computes and stores the bottom-k elements for each quantile in advance, so that when queries are executed or thresholds are updated, the system can work with these pre-processed sketches rather than processing raw streaming data from scratch. This preliminary action reduces the computational burden during updates and query execution.
Data Source
AI summary
The present disclosure relates to method for responding to a query requesting an intersection being performed. The method includes receiving a query referencing a first set, a second set, and a desired quantile related to the first set from among a plurality of quantiles; generating a data structure including a bottom-k sketch of user identifiers (ids) of the first set and corresponding numerical values of the first data; partitioning the data structure into a plurality of sketches to correspond to the quantiles, respectively; determining an intersection of one of the sketches associated with the desired quantile and a sketch of the second set; and responding to the query based on the intersection.