LLM Content Segment Selection Under Token Limits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face inefficiencies in locating specific content due to token limits in Large Language Models (LLMs), leading to manual searches and inefficient use of computing resources, particularly when dealing with large volumes of data across various formats.
Innovation Solution
A system that selects specific document segments based on token limits, time periods, and contextual relevance to generate summaries using LLMs, optimizing resource usage and maintaining context.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If LLMs are used to summarize large volumes of collaborative content, then content discovery accuracy is improved, but token limits prevent processing of complete documents
Solution Approach 1:
The system divides a large document into multiple segments, each within the LLM's token limit. It then processes each segment individually and combines the results, enabling summary of complete documents that would otherwise exceed processing capacity.
Solution Approach 2:
The system performs preliminary analysis to identify and select the most relevant segments before processing them through the LLM. This pre-filtering ensures that the limited tokens are used efficiently on content most likely to contain the information the user seeks.
2Adaptability or versatility
If manual searches are conducted to locate specific content, then flexibility in searching is maintained, but time and computational resources are wasted
Solution Approach 1:
The system introduces an intermediary layer between the user's search query and the LLM processing. This intermediary automatically identifies and retrieves relevant segments based on the query, eliminating the need for users to manually search through multiple sources while maintaining search flexibility.
Solution Approach 2:
The system enables self-service by automatically performing the content discovery task without requiring manual intervention. It autonomously identifies relevant segments, processes them through the LLM, and returns results, saving users time while maintaining adaptability through customizable search parameters.
3Reliability
If multiple queries are sent to locate information across different formats, then comprehensive coverage is achieved, but network and computational resources are consumed
Solution Approach 1:
The system creates a universal processing pipeline that handles multiple document formats (emails, chat threads, transcripts, etc.) through a single LLM interface. By standardizing the input format and using one versatile model, it eliminates the need for separate processing systems for each format, reducing computational overhead while maintaining comprehensive coverage.
Solution Approach 2:
The system merges multiple document formats and sources into a unified processing flow. It combines segments from emails, chat threads, transcripts, and other formats into a single coherent context for the LLM, achieving comprehensive information coverage through one integrated query rather than multiple separate ones.
4Loss of information
If LLMs process complete large documents, then context accuracy is maintained, but token limits are exceeded
Solution Approach 1:
The system performs preliminary analysis to identify and select the most contextually relevant segments before processing them through the LLM. This pre-filtering ensures that the limited tokens are used efficiently on content most likely to contain the information the user seeks, maintaining context accuracy without exceeding size limits.
Solution Approach 2:
The system divides a large document into multiple segments, each within the LLM's token limit. It then processes each segment individually and combines the results, enabling summary of complete documents that would otherwise exceed processing capacity while preserving contextual relationships through careful segment selection and ordering.
Data Source
AI summary
The disclosed techniques can overcome challenges that arise when large language models are used for summarizing documents that exceed the token limit of a LLM. The disclosed techniques communicate contextually relevant segments of a document with customized prompts to an LLM for generating a summary. A subset of segments can be selected from a document based one or more factors, including but not limited to, a token count of content of the select segments, a determination that the token count of content of the select segments does not exceed a token limit of the large language model. The subset of segments can also be selected based on a determination that the selected segments meet one or more criteria with respect to specific user activity. Thus, the system can also provide technical solutions that can use an LLM to discover content of interest stored in large files.