Boundary-Coalescing Text Chunks for Context-Rich LLM Responses
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing large language models (LLMs) struggle to provide contextually relevant responses due to limited grounding capabilities, which can be costly and inefficient when relying on multiple calls, and current chunking methods may not capture semantic meaning effectively.
Innovation Solution
The text data is divided into primary and boundary-coalescing secondary chunks, where secondary chunks span the boundaries between adjacent primary chunks, enhancing semantic similarity with conversational inputs to improve LLM grounding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If multiple calls are made to the LLM to provide grounding information, then the context coverage is improved, but the processing cost and time increase
Solution Approach 1:
The patent segments the grounding information retrieval into two distinct phases: an initial retrieval phase that captures broad context, and a subsequent boundary-coalescing phase that refines context by merging adjacent chunks. This segmentation allows the system to obtain comprehensive context coverage without requiring multiple separate LLM calls, as the boundary-coalescing operation is performed locally on the retrieved chunks rather than through additional LLM invocations.
Solution Approach 2:
The patent applies preliminary action by performing the boundary-coalescing operation on retrieved chunks before submitting them to the LLM. This pre-processing step consolidates adjacent text chunks that share boundary elements, ensuring that the LLM receives optimally structured context in a single call rather than requiring multiple calls to achieve the same contextual integration.
2Loss of information
If multiple calls are made to the LLM to provide grounding information, then the context coverage is improved, but the processing cost increases
Solution Approach 1:
The patent segments the grounding process into initial retrieval and boundary-coalescing phases, allowing comprehensive context to be obtained through a single LLM call followed by local processing. This eliminates the need for multiple expensive LLM calls while maintaining full context coverage, as the boundary-coalescing is performed efficiently on the retrieved chunks without additional LLM invocations.
Solution Approach 2:
The patent introduces an intermediary processing step (boundary-coalescing) that operates on the retrieved chunks before LLM submission. This intermediary operation consolidates adjacent chunks locally, serving as a cost-effective mediator that prepares optimized context for the LLM without requiring additional LLM calls, thereby reducing processing costs while maintaining context coverage.
3Productivity
If traditional chunking methods are used to divide text data, then the processing efficiency is improved, but the semantic meaning is lost at boundaries
Solution Approach 1:
The patent applies merging by combining adjacent text chunks at their boundaries to create coalesced chunks that preserve semantic meaning. When chunks are divided for processing efficiency, the boundary-coalescing operation merges overlapping portions of adjacent chunks, ensuring that semantic information at the boundaries is retained rather than lost, thus maintaining both processing efficiency and semantic integrity.
Solution Approach 2:
The patent applies local quality by applying different processing treatments to different portions of the text chunks. Specifically, boundary regions of adjacent chunks are coalesced together to preserve semantic meaning, while interior regions maintain their original chunked structure for processing efficiency. This localized quality enhancement ensures that semantic-critical boundary areas receive special treatment without compromising overall processing efficiency.
Data Source
AI summary
Database systems and methods are provided for managing usage of large language models (LLMs). One method involves dividing text data into primary chunks using input criteria associated with an LLM service, generating secondary chunks by merging respective pairs of adjacent primary chunks, and inputting a respective secondary chunk to the LLM service when a semantic similarity between a conversational input to a user interface and the respective secondary chunk of the one or more secondary chunks is greater than a threshold. The LLM service generates response data responsive to the conversational input based at least in part on a subset of the text data associated with the respective secondary chunk, and a response is provided to the conversational input at the user interface based at least in part on the response data generated by the LLM service.