Batched Prompt LLM Inference for Multi-User Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current large language model (LLM) implementations face challenges in efficiency, latency, complexity, and energy consumption, especially when handling a large number of users, making it difficult to operate efficiently at scale.
Innovation Solution
The proposed solution involves combining LLM queries from multiple users into a batched prompt, which is processed by a single LLM inference. This approach includes using prompt-engineering inputs to segment the output accurately and efficiently process the batched prompt, utilizing batching parameters such as temporal, spatial, and subject-matter parameters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If LLM queries from multiple users are processed individually, then each user receives personalized response, but system complexity and energy consumption increase significantly
Solution Approach 1:
The patent combines multiple LLM queries from different users into a single batched prompt that is processed by one LLM inference. The queries are structured with user identifiers and segmentation instructions, allowing the single LLM output to be divided and distributed to multiple users. This merging approach reduces system complexity and energy consumption while maintaining personalized responses for each user.
2Reliability
If LLM queries from multiple users are processed individually, then response quality is maintained, but energy consumption increases
Solution Approach 1:
Multiple user queries are merged into a single batched prompt processed by one LLM inference, reducing the total number of computational operations and energy consumption. The prompt includes segmentation instructions that ensure the output can be accurately divided and distributed to individual users, maintaining response quality while significantly reducing energy usage compared to processing queries separately.
3Ease of operation
If LLM queries from multiple users are processed individually, then each query is handled independently, but latency increases
Solution Approach 1:
The patent merges multiple independent queries into a single batched prompt that is processed simultaneously by one LLM inference. This parallel processing approach reduces the total time required to handle all queries compared to sequential individual processing. The structured prompt format with segmentation instructions ensures that each user receives their personalized response after the single LLM inference completes, reducing latency while maintaining query independence.
4Productivity
If queries are batched together, then processing efficiency improves, but output segmentation accuracy becomes challenging
Solution Approach 1:
The patent incorporates segmentation instructions as a preliminary action within the batched prompt itself, guiding the LLM to structure the output in a way that facilitates accurate segmentation. The prompt includes user identifiers and formatting instructions that pre-determine how the output should be organized, making the segmentation process more accurate and reliable without requiring complex post-processing.
5Productivity
If batched prompts are used, then number of LLM inferences is reduced, but prompt engineering complexity increases
Solution Approach 1:
The patent incorporates all necessary prompt engineering elements, including user identifiers, query formatting, and segmentation instructions, into the batched prompt itself as a preliminary action. This approach consolidates the complexity into a single structured prompt template that can be reused, reducing the need for complex prompt engineering for each individual query while maintaining high processing efficiency.
Data Source
AI summary
Disclosures are provided for execution of large-language models (LLMs), including systems and methods that allow for increased multi-user efficiency within an execution framework. For example, LLM queries provided by a plurality of users of an LLM may be combined together into a batched prompt that can be received by an LLM. In addition, the LLM receiving the batched prompt may be configured to provide an output that is segmented in accordance with each user's query. Configuration of the LLM may include providing the LLM with prompt-engineering inputs in connection with the batched prompt.