Partition-Based Histogram for String Selectivity Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for estimating selectivity in XML databases face challenges due to the large number of distinct string values at leaf nodes, making it difficult to accurately estimate the selectivity of string predicates, especially for exact and substring match queries.
Innovation Solution
A partition-based histogram is generated to support accurate selectivity estimation for both exact and substring match queries, using a three-step process involving path-value pair counting, partitioning, and synopsis extraction, which includes occurrence counts of paths and q-grams of values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional selectivity estimation methods are used for XML databases, then the system can process queries, but the estimation accuracy deteriorates due to the large number of distinct string values at leaf nodes
Solution Approach 1:
The patent segments the string values into buckets based on their frequency of occurrence. High-frequency strings are placed in one bucket while low-frequency strings are placed in another bucket. This segmentation allows the system to handle the large number of distinct string values by grouping them into manageable categories, thereby improving selectivity estimation accuracy without being overwhelmed by the quantity of individual string values.
Solution Approach 2:
The patent creates a simplified model (copy) of the string value distribution using histograms that represent the frequency distribution of strings in buckets. Instead of working with the actual large number of distinct string values, the system uses these histogram copies to perform selectivity estimation, which maintains accuracy while reducing the complexity of handling the full string value set.
2Measurement precision
If detailed string value statistics are maintained to improve estimation accuracy, then selectivity estimation improves, but the storage space and processing complexity increase
Solution Approach 1:
The patent segments string values into frequency-based buckets, which simplifies the statistics collection process. Instead of tracking every individual string value, the system only needs to maintain statistics for the buckets (e.g., number of strings in each bucket, average frequency), significantly reducing the complexity of statistics maintenance while preserving estimation accuracy.
Solution Approach 2:
The patent changes the parameters used for statistics from individual string values to bucket-level aggregates. Instead of storing detailed distributions of each string, the system stores aggregated parameters such as bucket frequencies and string count distributions, which are easier to collect and maintain while providing sufficient information for accurate selectivity estimation.
Data Source
AI summary
Histogram construction and selectivity estimation for string and substring match queries in databases of data having strings associated with attributes. The histogram construction counts string-attribute pairs in the documents, and outputs string-attribute-count triples sorted by count. The collection is partitioned into buckets. A synopsis is generated for the partition, having an average selectivity or count of the string-attribute-count triples in the partition and summary information representing the set of string-attribute pairs belonging to the bucket. Subsequent queries, both for exact and substring matches, use the synopsis to estimate the selectivity of buckets.