LLM Retrieval Index Partitioning for Large Dataset Query Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing information retrieval systems struggle with comprehending complex queries and managing computational and memory constraints when processing large datasets, leading to inefficient query response times and resource utilization.
Innovation Solution
An LLM-based system uses a pre-created index partitioned into voronoi cells to organize vector embeddings, allowing for efficient similarity searches by identifying relevant embeddings within a subset of cells, reducing search latency and resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional search engines process explicit queries using algorithms and ranking mechanisms, then they can retrieve relevant information, but they are limited in comprehending complex queries and face computational constraints when processing large datasets
Solution Approach 1:
The patent segments the large dataset into smaller partitions and creates separate indexes for each partition. This segmentation allows the system to process queries by only searching relevant partitions rather than the entire dataset, reducing computational complexity while maintaining the ability to handle complex queries across large datasets.
Solution Approach 2:
The patent performs preliminary actions by pre-computing vector embeddings for all documents and organizing them into partitions before query processing. This preliminary embedding and partitioning creates an optimized structure that enables faster query processing without requiring complex real-time computations, thus reducing device complexity requirements.
2Measurement precision
If the system creates a comprehensive index for all text files to ensure complete search coverage, then query accuracy improves, but search latency and resource usage increase
Solution Approach 1:
The patent divides the comprehensive index into multiple partitions based on vector embedding similarity. When a query is processed, the system identifies which partitions are most relevant and searches only those partitions rather than the entire index. This segmentation maintains query accuracy by focusing on relevant data while significantly reducing search latency and resource usage.
Solution Approach 2:
The patent applies partial action by searching only a subset of partitions that are most relevant to the query, rather than performing a complete search of all partitions. This partial search approach maintains sufficient query accuracy by targeting the most promising partitions while reducing overall search time and computational resources required.
3Quantity of substance
If the system processes large datasets with high dimensionality, then comprehensive information is retained, but computational efficiency and memory usage deteriorate
Solution Approach 1:
The patent segments the high-dimensional data space into multiple partitions based on vector embedding characteristics. This segmentation allows the system to process and store large datasets more efficiently by organizing information into manageable chunks, maintaining information retention while improving computational efficiency and reducing memory usage during query processing.
Data Source
AI summary
Certain aspects of the disclosure provide techniques for information retrieval for large datasets. A method comprises receiving input text files and a query for the files; obtaining an index associated with the input text files to process the query, wherein: the index comprises key-value mappings, each key of a respective mapping identifying a voronoi cell of the index, and each value of a respective mapping identifying vector embeddings associated with text files associated with a voronoi cell of the index; creating a query embedding based on the query; identifying a first key-value mapping having a first key associated with a first voronoi cell in the index and corresponding to the query embedding; obtaining a set of vector embeddings associated with the first value; comparing the query embedding to the set of vector embeddings to determine closest vector embeddings; and generating a textual output based on the closest vector embeddings.