Split Document Indexing for Generative AI Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing generative machine learning systems face challenges in efficiently accessing and utilizing diverse data sources to provide accurate and relevant responses to natural language requests, leading to poor performance and potential hallucinations.
Innovation Solution
Implementing indexing split documents for data retrieval, which involves splitting documents into portions, creating efficient search indexes, and using retrieval augmented generation techniques to enhance the performance of generative machine learning systems by optimizing computing resources and providing relevant data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If generative machine learning systems access diverse data sources directly, then data completeness is improved, but system performance and response accuracy deteriorate due to inefficient data access and potential hallucinations
Solution Approach 1:
The patent segments documents into smaller chunks or portions before indexing them. This segmentation allows the system to efficiently retrieve only relevant portions rather than entire documents, improving both data access efficiency and response accuracy while maintaining data completeness.
Solution Approach 2:
The patent implements preliminary indexing of document portions before actual queries are made. By pre-processing and organizing document segments into searchable indexes, the system enables fast retrieval of relevant data during inference, resolving the contradiction between data completeness and response accuracy.
2Quantity of substance
If documents are indexed in full, then data completeness is improved, but computing resource efficiency deteriorates
Solution Approach 1:
Instead of indexing entire documents, the patent divides documents into smaller portions or chunks. This segmentation reduces the overall indexing workload and storage requirements while maintaining the ability to retrieve complete information through multiple smaller index queries.
Solution Approach 2:
The patent extracts only the necessary portions of documents for indexing rather than processing complete documents. This extraction approach reduces computing resources required for indexing while preserving data completeness through selective retrieval of relevant segments.
3Quantity of substance
If multiple data sources are checked to locate desired information, then data completeness is improved, but time consumption increases
Solution Approach 1:
The patent performs preliminary indexing of document portions from multiple data sources before actual information retrieval is needed. This pre-processing creates searchable indexes that enable fast querying across multiple sources simultaneously, reducing the time required to locate desired information while maintaining data completeness.
Solution Approach 2:
The patent introduces an intermediary indexing layer that sits between multiple data sources and the query system. This intermediary organizes document portions into a unified searchable structure, enabling efficient cross-source information retrieval without checking each source individually.
Data Source
AI summary
An index is created with split documents to retrieve and augment generation of a response to a natural language request using a generative machine learning model. When a natural language request is received, a search representation is generated and used to retrieve candidate portions of documents from the index. A relevancy ranking is performed to identify relevant portions of documents from the candidates and provide the relevant portions to prompt a generative machine learning model to provide a result for the natural language request.