Hash Filter Column Indexing With Truncated Values
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database systems face inefficiencies in accessing and updating column values, particularly in terms of memory overhead and processing resources, especially when using traditional indices, which can lead to suboptimal performance in search operations and resource utilization.
Innovation Solution
The proposed solution involves encoding database table columns as arrays of hash values and reduced strings, allowing for quick scanning to determine the existence of values and efficiently locating row IDs without significant memory overhead, using a k-byte representation and truncated values to reduce storage size and improve search speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional indices are used to enable fast determination of rows containing particular column values, then search speed is improved, but memory overhead increases significantly
Solution Approach 1:
The patent extracts only the most essential indexing information (first k bytes of column values) to create a compact filter structure, discarding redundant data while maintaining search capability. This selective extraction reduces memory overhead significantly compared to traditional indices that store complete or near-complete column value representations.
Solution Approach 2:
The patent applies different storage qualities to different parts of the data: the first k bytes are stored in a compact, space-efficient format in the filter structure, while the remaining bytes are stored in the original column storage. This local differentiation optimizes memory usage by applying high compression where possible and preserving original data where needed for accuracy.
2Speed
If traditional indices are used to enable fast determination of rows containing particular column values, then search speed is improved, but processing resources increase significantly
Solution Approach 1:
The patent segments the indexing process into two distinct phases: a compact filter structure for rapid initial screening that consumes minimal processing resources, and a secondary verification stage that only processes rows passing the filter. This segmentation prevents the system from expending full processing resources on every row, significantly reducing overall energy consumption while maintaining search speed.
Solution Approach 2:
The patent performs a partial indexing action by only hashing and storing the first k bytes of column values rather than complete values. This partial action is sufficient for the primary filtering function while consuming fraction of the processing resources required for complete value indexing.
3Measurement precision
If column values are checked entirely to determine whether a particular value is stored, then accuracy is maintained, but search speed decreases
Solution Approach 1:
The patent performs preliminary filtering using the compact hash filter structure before conducting full value comparisons. This preliminary action eliminates the majority of non-matching rows early in the search process, allowing the system to maintain high search accuracy while achieving fast search speeds by avoiding unnecessary full comparisons.
Solution Approach 2:
The patent introduces a hash filter structure as an intermediary between the query and the full column data. This intermediary performs initial matching based on hashed first k bytes, providing a fast approximation that guides subsequent precise matching operations, thereby balancing speed and accuracy.
Data Source
AI summary
A system includes determination of a first value of a first row of a database table column, generation of a first truncated value based on the first value, the first truncated value including fewer bytes than the first value, generation of a first representation of the first value based on the first truncated value, the first representation including fewer bytes than the first value, storage of the first representation at a first position in a first array, and storage of the first truncated value at a second position in a second array.