Hilbert Curve Mapping for Semantic Embedding Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Efficiently searching and retrieving relevant documents from large databases is challenging due to high time complexity and computational overhead, especially when matching longer semantic embedding vectors, which hinders quality and efficiency in document retrieval.
Innovation Solution
The method employs the Hilbert curve to map multidimensional semantic embedding vectors into one-dimensional vectors, creating an index table and using binary search to reduce time complexity and computational overhead while maintaining high-quality search results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If sequential matching of longer embedding vectors is used to ensure semantic matching quality, then search accuracy is improved, but search time increases significantly
Solution Approach 1:
The patent applies dimensionality change by mapping multi-dimensional semantic embedding vectors to one-dimensional Hilbert numbers through the Hilbert curve transformation. This allows the system to preserve semantic relationships while enabling efficient one-dimensional binary search, resolving the contradiction between search accuracy and search time by operating in a transformed dimensional space.
Solution Approach 2:
The Hilbert curve acts as an intermediary transformation that bridges the gap between multi-dimensional semantic embedding space and one-dimensional search space. By introducing this intermediate mapping layer, the system maintains the semantic matching quality of the original embedding vectors while enabling fast binary search operations on the transformed one-dimensional Hilbert numbers.
2Measurement precision
If semantic embedding vectors for sentences are used instead of entire document embedding vectors, then search quality is improved, but computational overhead increases
Solution Approach 1:
The patent reduces computational overhead by transforming sentence-level semantic embedding vectors into compact one-dimensional Hilbert numbers. This dimensionality change compresses the representation while preserving semantic information, allowing the system to process multiple sentence embeddings efficiently without the quadratic complexity of comparing long embedding vectors directly.
Solution Approach 2:
The patent segments documents into sentences and processes each sentence's embedding vectors separately, transforming them into Hilbert numbers. This segmentation allows the system to handle large documents by processing smaller units independently, reducing the computational overhead compared to processing entire document embeddings as a single large vector.
3Speed
If Hilbert curve mapping is applied to accelerate search, then search speed is improved, but system complexity increases
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing the Hilbert curve mappings of embedding vectors in an index structure during an offline phase. This preprocessing step transforms the complex multi-dimensional vectors into one-dimensional Hilbert numbers in advance, so that during online search, the system only needs to perform simple binary search operations, achieving fast search speed with minimal online computational complexity.
Data Source
AI summary
A system and method includes a textual document search engine. The method initializes the textual document search engine by inputting, into a memory, a very large number of documents, where each document has textual sentences, and each sentence of each document has m semantic embedding vectors. The method maps the m semantic embedding vectors to 1-dimensional vectors of Hilbert numbers using a Hilbert curve transformation, constructs an index table with the 1-dimensional vectors, and stores the index table in the memory. The index table is used for efficient search by inputting a full query document, which has embedding vectors corresponding to sentences in the query document, where the query embedding vectors are mapped into 1-dimensional vectors of Hilbert numbers using Hilbert curve transformation. The index table is searched using the Hilbert numbers and candidate documents are retrieved that are similar to the query document based on the Hilbert numbers.