Dataset Summary Metadata Using Gap Filters for Fast Value Checks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing sorting techniques and summary information methods are computationally expensive and require large storage, leading to inefficiencies in determining whether a particular value is present in a dataset, and they can result in false positive outcomes.
Innovation Solution
The use of a gap filter, which represents a sorted set of unique values with gaps, allowing for efficient determination of value presence through a compressed representation that balances compressibility and false positive rates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional sorting techniques are used to create summary information, then the accuracy of determining value presence is improved, but the computational cost and time required increase significantly
Solution Approach 1:
The patent segments the dataset into multiple partitions or blocks, creating summary information (min/max values) for each partition independently. This allows parallel processing of partitions during query evaluation, reducing the overall time required while maintaining accurate determination of value presence across the entire dataset.
Solution Approach 2:
The patent pre-computes and stores summary information (minimum and maximum values) for dataset partitions before query execution. This preliminary action enables fast query evaluation by allowing the system to quickly determine whether a queried value could potentially exist in a partition without scanning the actual data, thus reducing query processing time while maintaining accuracy.
2Measurement precision
If detailed summary information is stored for all datasets, then the ability to accurately determine value presence is improved, but the storage requirements increase significantly
Solution Approach 1:
The patent applies local quality by creating summary information only for specific partitions or blocks of data rather than for the entire dataset uniformly. This allows the system to store minimal summary information (just min/max values) locally for each partition, reducing overall storage requirements while maintaining the ability to accurately determine value presence through selective partition evaluation.
Solution Approach 2:
The patent extracts only the essential summary information (minimum and maximum values) from each data partition, discarding the rest of the detailed data for storage purposes. This extraction approach maintains the ability to determine value presence accuracy while significantly reducing the quantity of stored summary information compared to storing complete dataset details.
3Reliability
If comprehensive summary information is created for every dataset, then false positives are reduced, but the computational resources required to create and maintain this information increase
Solution Approach 1:
The patent applies partial action by creating summary information for only certain partitions or a subset of datasets rather than comprehensively for every dataset. This selective approach reduces the computational resources required to create and maintain summary information while still achieving acceptable false positive rates for queries that can be evaluated using the available summary information.
Data Source
AI summary
The present disclosure provides techniques and solutions for sorting data. In a particular implementation, a sorting technique is provided that places values in a sorted order by adding an offset value to values that are not in a sorted order. The resulting sorted set of values is not truly sorted, in that the set of modified values is sorted, but the underlying data itself is not sorted. In another implementation, a sorting technique can use multiple streams or sets. When an out of order element is encountered, it can be added to a new stream, if such a stream is available. The sorting techniques can be used for a variety of purposes, including provided sorted data for use in generating summary data, or for providing sorted data to be used in determining an intersection between two datasets.