LLM Orchestrator Agent for API Query Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) face difficulties in providing accurate and comprehensive responses in multi-service big platform systems due to limitations in understanding domain-specific jargon and context, leading to truncated outputs and inaccurate responses when dealing with lengthy API endpoint documentation and complex user queries.
Innovation Solution
An orchestrator agent coordinates with service agents equipped with LLMs to formulate and execute processing plans, including API calls, to manage tasks across multiple services, ensuring accurate and coherent responses by breaking down queries into manageable prompts and handling intricate data sources and APIs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If LLMs process lengthy API endpoint documentation and complex user queries directly, then comprehensive context understanding is required, but token limits cause truncated outputs and inaccurate responses
Solution Approach 1:
The patent segments lengthy API documentation and complex queries into smaller, manageable prompt components. The system divides the input into distinct segments (user query, relevant API documentation snippets, context information) that can be processed within token limits while preserving essential meaning and context.
Solution Approach 2:
The patent extracts only the most relevant portions of API endpoint documentation needed to answer specific user queries. Instead of including complete documentation, the system identifies and extracts specific API parameters, endpoints, and contextual information that directly relate to the user's question, reducing token usage while maintaining accuracy.
2Measurement precision
If LLMs are provided with extensive API documentation to ensure accurate responses, then response accuracy improves, but processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary actions by pre-processing and organizing API documentation into structured formats before LLM processing. The system pre-identifies relevant API endpoints, parameters, and examples that may be needed, preparing them in advance so that during query processing, only relevant pre-processed information needs to be included in prompts, reducing processing time while maintaining accuracy.
3Reliability
If LLMs handle complex multi-service platform queries with domain-specific jargon, then comprehensive domain understanding is required, but the model produces truncated and inaccurate outputs
Solution Approach 1:
The patent introduces an intermediary layer that translates domain-specific jargon and complex queries into LLM-friendly formats. The system acts as a mediator by parsing domain-specific language, mapping it to standard API terminology, and structuring information in ways that preserve domain context while ensuring the LLM can process and generate accurate responses.
Data Source
Figure 1
Figure 2A~2B
Figure 3
AI summary
Systems and methods for incremental solves using LLMs for API calls is presented. The systems and methods produce, by a first large learning model (LLM), a processing plan based on a first prompt, wherein the processing plan includes a plurality of tasks corresponding to a plurality of services. The systems and methods send a plurality of messages corresponding to the plurality of tasks to a plurality of service agents, wherein the plurality of service agents correspond to the plurality of services and comprise a plurality of second LLMs that produce a plurality of agent responses. The systems and methods then generate a query response based on the plurality of agent responses.