Semantic Chunk Selection for Non-Redundant AI Query Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional approaches to document chunking for generative AI models like LLMs fail to address multiple potential answers from different text chunks, leading to redundant information and omission of important relevant information due to prompt size limitations.
Innovation Solution
A method that generates chunk embeddings, determines non-overlapping content between candidate chunks, and selects a subset based on similarity and non-overlapping content for inclusion in a prompt to the generative AI model, ensuring optimal chunk selection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional successive chunking is used to fit prompt size limitations, then the prompt can accommodate more content, but multiple potential answers from different text chunks are omitted and redundant information is presented
Solution Approach 1:
The document is divided into multiple chunks of appropriate size for prompt input, allowing the system to process large documents while maintaining prompt size constraints. This segmentation enables inclusion of multiple potential answers from different text chunks without exceeding prompt limits.
Solution Approach 2:
The system transitions from simple sequential chunking to a multi-dimensional selection approach by evaluating chunks based on multiple criteria including semantic similarity to query, overlap analysis with previously selected chunks, and relevance scoring. This dimensional expansion of the selection process enables optimal chunk choice that maximizes information coverage while minimizing redundancy.
2Device complexity
If conventional successive chunking is used, then prompt size limitations are respected, but redundant information is presented to the user
Solution Approach 1:
Before final prompt construction, the system performs preliminary analysis by calculating semantic similarities between all candidate chunks and the query, and determining overlap relationships between pairs of chunks. This preliminary computation enables the system to select non-redundant chunks that provide diverse information while respecting prompt size constraints.
Solution Approach 2:
The system incorporates feedback loops where the selection of each chunk influences subsequent chunk selections. By analyzing overlap between previously selected chunks and evaluating remaining candidates, the system dynamically adjusts selections to avoid redundancy while maximizing information content within the prompt.
3Ease of manufacture
If conventional successive chunking is used, then processing is simple, but important relevant information is omitted due to prompt size limitations
Solution Approach 1:
The system changes the parameters of chunk selection from simple sequential ordering to a multi-criteria evaluation system. By introducing parameters such as semantic similarity score, overlap ratio, and relevance weighting, the system can reliably identify the most important information while maintaining manageable processing complexity through automated calculations.
Data Source
AI summary
A method, computer program product, and computing system for generating a plurality of chunks for a plurality of text portions of a document. A plurality of chunk embeddings are generated from the plurality of chunks. A query is processed using a generative artificial intelligence (AI) model. A query embedding is generated from the query. A plurality of candidate chunks are identified from the plurality of chunks based upon, at least in part, a similarity between the plurality of chunk embeddings and the query embedding. An amount non-overlapping content of each candidate chunk is determined relative to each other candidate chunk. A subset of the plurality of candidate chunks are selected for inclusion in a prompt with the query based upon, at least in part, the amount of non-overlapping content of each candidate chunk.