Probabilistic Data Structures for Time-Range Query Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computing systems, efficiently querying log data across multiple data cores for specific time ranges is challenging due to the need to search every data core, which is time-consuming and expensive, especially when minimum and maximum time values overlap, making it difficult to distinguish between data cores.
Innovation Solution
The use of probabilistic data structures, such as Bloom filters, to determine a subset of data cores to query based on a specified time range, reducing the need to search every data core by indexing reference time values and using a predetermined time resolution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If every data core is searched for log data at a specific time range, then all relevant data can be found, but the search becomes time-consuming and expensive
Solution Approach 1:
The system pre-computes and stores minimum and maximum time values for each data core before queries are executed. This preliminary indexing allows the query system to quickly identify candidate data cores without searching every core, thus reducing search time while maintaining the ability to find all relevant data.
Solution Approach 2:
The patent introduces an intermediary metadata layer (minimum and maximum time value indices) that mediates between the query request and the actual data cores. This intermediary structure enables efficient filtering of candidate data cores before the actual search, resolving the contradiction between comprehensive search and search efficiency.
2Quantity of substance
If minimum and maximum time values are stored for each data core to reduce metadata, then the amount of metadata is reduced, but time range distinguishability between data cores is lost when ranges overlap
Solution Approach 1:
The system changes the parameter representation from storing individual data item timestamps to storing aggregated minimum and maximum time values. This parameter transformation reduces metadata size while maintaining sufficient precision for time-range query routing, resolving the contradiction between metadata compactness and query accuracy.
3Measurement precision
If a brute force approach maintains a list of timestamps for every data item in each data core, then accurate mapping is achieved, but the approach is neither scalable nor efficient
Solution Approach 1:
The patent extracts only the essential temporal boundary information (minimum and maximum time values) from the complete set of data item timestamps. This extraction eliminates the need to store and process every individual timestamp while retaining sufficient information for accurate data core identification, thus improving scalability without sacrificing mapping accuracy.
Data Source
AI summary
The present disclosure relates to querying data cores for data items that correspond to a specified time range. Probabilistic data structures corresponding to associated data cores are used to filter the plurality of data cores to identify a subset of data cores that contain data items corresponding to the specified time range. Only the subset of the plurality of data cores determined to contain relevant data items are searched.