Parallel Nested LLM Prompt Processing for Faster Response Streaming
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) face processing delays due to the analysis of system prompts provided by service provider platforms, which are not visible to users but slow down response times when combined with user input prompts.
Innovation Solution
Implementing a system and method for parallel and nested processing of prompts, where token groups from system and user prompts are analyzed independently or conditionally, using a parallel processing engine and nested loop engine to reduce processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If system prompts are provided to guide and shape LLM responses, then response quality is improved, but processing time increases
Solution Approach 1:
The patent segments the prompt into multiple token groups and processes them in parallel across multiple LLM instances. Each token group is handled independently by different models, allowing simultaneous processing of system prompts and user input prompts, thereby reducing overall processing time while maintaining response quality through coordinated results aggregation.
Solution Approach 2:
The patent transitions from sequential processing (single dimension) to parallel processing across multiple LLM instances (multi-dimensional). By distributing token groups across different model instances that process simultaneously, the system achieves time compression while maintaining the comprehensive analysis required for quality responses.
2Device complexity
If prompts are processed sequentially, then processing simplicity is maintained, but response time increases
Solution Approach 1:
The patent divides the prompt into distinct token groups (system prompts and user input prompts) that can be processed independently. This segmentation enables parallel processing architecture where multiple LLM instances handle different token groups simultaneously, reducing response time while managing complexity through structured organization of processing tasks.
Solution Approach 2:
The system performs preliminary tokenization and grouping of prompts before processing. By preorganizing the prompt into token groups that are ready for parallel processing, the system minimizes processing complexity during execution while achieving faster response times through concurrent processing of prepared token groups.
3Device complexity
If all token groups are processed together, then processing is simplified, but processing efficiency decreases
Solution Approach 1:
The patent segments token groups into independent units that can be processed concurrently. System prompts and user input prompts are separated into distinct token groups that are distributed to multiple LLM instances for simultaneous processing, thereby improving productivity while maintaining reasonable processing structure through organized task distribution.
Solution Approach 2:
The patent processes only the necessary token groups in parallel rather than all tokens sequentially. By identifying and processing independent token groups concurrently, the system achieves higher processing efficiency without requiring complete processing of all token groups simultaneously, optimizing the balance between processing structure and productivity.
Data Source
AI summary
A system and method processes token groups input to an LLM in parallel and/or by nested processing. Each token group may consist of one or more tokens from a system prompt and user prompt. In addition to simple parallel processing of the one or more token groups, prompts may be input as nested prompts, where processing of one or more token groups may be begin and end at different times, depending on satisfaction of a start and/or end condition.