Compressed Hash Table Indexing for Low Latency Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Searching through large hash tables can be time-consuming and processor-intensive, often requiring storage in slower memory due to their size, which increases search time.
Innovation Solution
A processor-based method that breaks values into address bits, prefix bits, and signature bits to determine a container using a compressed index, comprising a prefix table and a signature table, to efficiently search for values in a hash table.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If large hash tables are stored in slower memory to accommodate their size, then storage capacity is improved, but search time increases
Solution Approach 1:
The hash table is divided into multiple containers, each containing a prefix table and signature table. The prefix table stores compressed prefix information while the signature table stores signature bits. This segmentation allows the system to store large hash tables in slower memory while enabling fast searches by only loading and processing relevant portions (containers) into faster memory, thus resolving the contradiction between storage capacity and search time.
2Quantity of substance
If large hash tables are stored in slower memory, then storage capacity is improved, but processor cycle intensity increases
Solution Approach 1:
The invention extracts only the essential components needed for searching (prefix tables and signature tables) from the full hash table and stores them in containers. When searching, only the specific container containing the target key needs to be loaded into faster memory and processed, rather than loading the entire hash table. This extraction approach reduces processor cycle intensity while maintaining storage capacity for large hash tables in slower memory.
3Productivity
If compressed index with prefix and signature tables is used, then search efficiency is improved, but device complexity increases
Solution Approach 1:
The hash table is pre-processed during construction to generate prefix tables and signature tables for each container. Prefixes are computed and stored in the prefix table, and signature bits are prepared in the signature table. This preliminary action enables efficient searching by allowing the system to quickly determine which containers to search and how to interpret results, improving search efficiency while managing complexity through structured pre-computation.
Data Source
AI summary
Methods and apparatus for determining if a value is stored in a hash table is provided. The method includes breaking the value into address bits, prefix bits, and signature bits. The method includes determining a container in a compressed index at an address specified by the address bits, the container comprised of a prefix table with bits set corresponding to the prefix bits determined by aggregate values associated with the container and a signature table containing the signature bits determined by the aggregate values associated with the container. The method includes determining a result based on a function of the prefix and signature tables and the determined prefix and signature bits.