Probabilistic Top-K Querying for Streamed Data Bins
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods and systems cannot effectively handle top-k queries for arbitrary, selectable intervals in large, streamed data sets, particularly when using probabilistic solutions, as they are limited to infinite time intervals and do not allow for querying of top-k information within specific time intervals.
Innovation Solution
A computer-implemented method and system that utilize a data structure with bins associated with unique time intervals, storing entries with maximum and minimum possible count values, which are updated using a probabilistic algorithm to account for errors, allowing for querying and determining top-k data within any requested time interval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional probabilistic solutions are used to store event data, then storage efficiency is improved, but the ability to handle queries for arbitrary time intervals deteriorates
Solution Approach 1:
The data structure is divided into multiple bins, where each bin corresponds to a specific time interval. This segmentation allows the system to efficiently store data in a compact probabilistic format while enabling queries for any arbitrary time interval by selecting and combining the relevant bins. The segmentation principle resolves the contradiction by organizing data spatially and temporally so that both storage efficiency and query flexibility are achieved simultaneously.
2Device complexity
If top-k information is provided for infinite time intervals only, then system simplicity is maintained, but query flexibility for specific time intervals deteriorates
Solution Approach 1:
By segmenting the data structure into time-interval-based bins, the system maintains relative simplicity while enabling flexible queries. Each bin is independently manageable, and the modular design allows the system to handle both infinite and finite time interval queries without requiring a complete redesign of the data structure.
Solution Approach 2:
The bin-based data structure serves multiple functions: it stores data for infinite time intervals (maintaining compatibility with existing simple queries) and simultaneously enables queries for arbitrary finite time intervals by allowing selective combination of bins. This multi-functionality resolves the contradiction by making the system adaptable to different query types without sacrificing simplicity.
3Productivity
If probabilistic algorithms are used to determine count values, then processing speed is improved, but measurement precision of count values deteriorates
Solution Approach 1:
The system预先 computes and stores error values alongside the probabilistic count estimates in each bin. These error values act as a cushion that quantifies the uncertainty in the probabilistic measurements. When answering queries, the system combines bins and aggregates both the count estimates and their associated errors, providing results with known precision bounds. This beforehand cushioning allows the system to maintain fast probabilistic processing while providing reliable precision information for decision-making.
Data Source
AI summary
A system and method are provided for enabling querying of a large set, including accessing a data structure associated with a metadata parameter and configured to store partial information associated with the data set in a plurality of bins. Each bin, associated with a unique time interval, is configured to store a plurality of entries associated with identified respective members of the metadata parameter's that have a detection time included in the bin's time interval. Each entry has at least one of an updated maximum and minimum possible count value determined using a probabilistic algorithm. The method includes receiving a query having a requested time interval, selecting two or more bins f the data structure that in combination describe the requested time interval, selecting k entries from a combination of the entries in the selected bins based on at least one of an updated maximum and minimum possible count value associated with entries of the selected bins, and determining top-k data, the top-k data including identification of the selected k entries.