Clu-Histogram Vector Representation for High-Dimensional Query Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional low-dimension histogram approaches are inefficient for high-dimensional databases, requiring excessive memory, storage, and computation resources, and there is a lack of effective solutions for query optimization in such scenarios.
Innovation Solution
The development of a Clu-Histogram (Clustered Histogram) method, which aggregates buckets into clusters represented as vectors in hyper-space, using clustering algorithms and linear regression to reduce the number of buckets and improve memory and computation efficiency, allowing for efficient query processing in high-dimensional data environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional low-dimension histogram approach is used, then selectivity estimation capability is provided, but memory and storage requirements become excessively large for high-dimensional databases
Solution Approach 1:
The patent segments the high-dimensional histogram buckets into multiple partitions or groups, where each partition covers a specific range of dimension values. This segmentation allows the system to store and process histogram data in smaller, more manageable units, reducing the memory and storage requirements while maintaining the ability to provide accurate selectivity estimates for queries by accessing only relevant partitions.
Solution Approach 2:
The patent introduces a new dimension or organizational structure to the histogram data by creating a hierarchical or multi-level partitioning scheme. Instead of storing all high-dimensional buckets in a flat structure that requires excessive memory, the data is reorganized into multiple dimensions of indexing and access, allowing efficient storage and retrieval without requiring proportional memory growth with increasing data dimensions.
2Measurement precision
If traditional low-dimension histogram approach is used, then selectivity estimation is available, but computation costs become unacceptable
Solution Approach 1:
The computation of selectivity estimates is segmented by partitioning the histogram access into multiple smaller operations. Instead of processing all high-dimensional buckets in a single computation, the system divides the work across multiple partitions, reducing the computational burden of each individual operation and enabling more efficient use of processing resources.
Solution Approach 2:
The patent performs preliminary organization and indexing of histogram buckets into partitions before query processing. This preliminary action prepares the data structure in advance, so that during query execution, the system can quickly locate and access only the relevant partition(s) without performing unnecessary computations on unrelated data, thereby reducing overall computation costs.
3Loss of information
If high-dimensional histogram is implemented, then comprehensive data distribution statistics are obtained, but device complexity increases significantly
Solution Approach 1:
The complex high-dimensional histogram structure is segmented into multiple simpler partition units, each managing a subset of the dimensional space. This segmentation reduces the apparent complexity of the overall structure by breaking it down into manageable components with simpler internal organization, while still capturing comprehensive data distribution statistics across all dimensions through the collective coverage of partitions.
Solution Approach 2:
The patent applies dimensional transformation by organizing histogram data along multiple hierarchical levels or access dimensions. This reorganization reduces the complexity of directly managing high-dimensional buckets by introducing intermediate indexing layers, making the overall structure more manageable while preserving the ability to represent comprehensive multi-dimensional data distribution.
Data Source
AI summary
Relational data for a database table having multiple columns (dimensions) is represented as a histogram. The buckets of the histogram are clustered into clusters, the clusters are fewer in number than the buckets of the histogram. Each cluster is represented in hyper-space as a line segment (vector) with a thickness value. Average frequencies for the line segments (vectors) and their thickness values are computed. The average frequencies are processed by a query optimizer when processing a query predicate of a query to use as an estimation of selectivity or an estimation of a cardinality of a join operation.