Hierarchical Histogram Mapping for Multidimensional Database Sorting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face inefficiencies in processing analytical queries due to the aggregation of data with significantly different values in the same storage locations, leading to increased data read amounts and query response times, particularly when handling non-integer data types.
Innovation Solution
A database management apparatus constructs a hierarchical histogram for each column, converting data to integer values using a multidimensional sorting algorithm, ensuring balanced data placement by maintaining high cardinality and preventing data with significantly different values from being aggregated in the same segment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If data is converted to integer values using only upper bits, then conversion simplicity is improved, but sorting granularity becomes coarse and data with significantly different values are aggregated together
Solution Approach 1:
The patent segments the data conversion process into multiple stages: first converting to intermediate integer values using upper bits, then further segmenting based on histogram analysis to achieve fine-grained sorting. This multi-stage segmentation resolves the contradiction by maintaining simplicity in the first stage while achieving precision in subsequent stages.
Solution Approach 2:
The patent introduces a histogram-based dimensional transformation that maps data values to integer bins. By adding this intermediate dimensional layer between simple bit truncation and final sorting, the system achieves both conversion simplicity and sorting precision without directly compromising either aspect.
2Stability of the object's composition
If value range widths of bins are adjusted to equalize data distribution, then balance of data allocation is improved, but data with significantly different values may be allocated to the same bin
Solution Approach 1:
The patent applies local quality by creating histograms with different bin width characteristics for different data regions. Rather than using uniform bin widths throughout, the system allows bin widths to vary locally based on data density and value ranges, maintaining both distribution balance and value differentiation precision in their respective regions.
Solution Approach 2:
The system dynamically changes the bin width parameter based on data characteristics. By adjusting bin width as a variable parameter rather than a fixed value, the patent achieves both balanced data allocation and precise value differentiation, resolving the contradiction between these two requirements.
3Productivity
If data is aggregated by multidimensional sorting, then query processing speed is improved, but data read amount increases when data with different values are colocated
Solution Approach 1:
The patent performs preliminary data placement optimization before query execution by pre-sorting data using multidimensional sorting based on histogram-derived integer values. This preliminary action ensures that data with similar values are physically collocated in advance, enabling efficient query processing while minimizing the actual data read amount during query execution.
Solution Approach 2:
The patent replaces mechanical data scanning with a computational histogram-based sorting mechanism. By substituting the mechanical process of reading and filtering data with a pre-computed integer value mapping system, the patent achieves both fast query processing and reduced data read amounts through intelligent data placement.
Data Source
AI summary
A database management apparatus constructs, for each column in input data, a hierarchical histogram of data distribution with respect to the column by repeating division into a prescribed number of areas based on a degree and a base suitable for a multidimensional sorting algorithm and creation of an equal-width histogram as long as an empty bin is present in a histogram, and creates integer value conversion data that maps a value range width of data in the input data to a converted integer value on the basis of the hierarchical histogram. The database management apparatus places the data, which is in the input data, in a database by multidimensionally sorting the input data according to the multidimensional sorting algorithm on the basis of the integer value conversion data of each column.