ValueID Lookup Table for Faster Dictionary Vector Scans
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Dictionary-compressed columns in high-speed in-memory database management systems require efficient search methods to reduce memory and processing resource usage, as large index vectors can lead to bandwidth limitations and intensive CPU cycles, especially in multi-socket systems.
Innovation Solution
Implement a method that logs the row number of the first appearance of each ValueID in the index vector, allowing scans to start from a specific position rather than the beginning, and uses a ValueID lookup table with group ranges to reduce memory overhead, enabling efficient searches by mapping ValueIDs to their starting positions in the index vector.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If scanning starts from the beginning of the index vector, then all ValueIDs are checked ensuring complete search coverage, but scanning time and CPU cycles increase significantly
Solution Approach 1:
The patent pre-computes and stores the starting position of each ValueID in a lookup table during data loading or maintenance operations. This preliminary action allows the search phase to skip directly to relevant positions without scanning from the beginning, thus resolving the contradiction between complete search coverage and scan speed.
2Productivity
If a lookup table storing starting positions for all ValueIDs is maintained, then search efficiency improves significantly, but memory overhead increases
Solution Approach 1:
The patent segments the lookup table into multiple smaller tables, each covering a specific range of ValueIDs. This segmentation reduces the memory footprint of each individual table while maintaining the ability to provide fast lookups for any ValueID by selecting the appropriate segment, thus resolving the contradiction between search efficiency and memory overhead.
3Reliability
If the index vector is fully scanned to ensure no occurrences are missed, then search accuracy is maintained, but bandwidth consumption increases in multi-socket systems
Solution Approach 1:
The patent extracts the starting position information from the index vector and stores it separately in a lookup table. This extraction allows the search process to access starting positions directly without scanning the entire index vector, thereby maintaining search accuracy while significantly reducing bandwidth consumption in multi-socket systems.
4Quantity of substance
If dictionary compression is applied to reduce memory usage, then main memory utilization improves, but search operation complexity increases
Solution Approach 1:
The patent introduces a lookup table as an intermediary structure that maps ValueIDs to their starting positions in the compressed index vector. This intermediary simplifies the search operation by providing direct access points, thereby reducing search operation complexity while maintaining the memory efficiency benefits of dictionary compression.
Data Source
AI summary
A ValueID corresponding to a value to be searched for in a database column can be determined by finding a match in a dictionary for the database column. A row of an index vector for the database column at which to begin a scan for the ValueID can be identified by reading a ValueID lookup table that maps each unique ValueID to a starting position in the index vector for the database column. The ValueID can not occur in the index vector prior to the starting position. The index vector can be scanned beginning at the starting position to find at least one occurrence of the ValueID corresponding to the value being searched, and a result that includes one or more row locations in the database column corresponding to the at least one occurrence in the index vector of the ValueID corresponding to the value being searched can be returned.