Recursive RAG Copilot Microservices for Faster, More Accurate Responses
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large Language Models (LLMs) face challenges with high computational burdens, latency, and undesirable artifacts such as inaccuracies, biases, and hallucinations, limiting their practical application in various domains.
Innovation Solution
A microservice architecture comprising small to mid-sized trained machine learning tools and software modules, allowing for efficient deployment on single-node systems, customized training, and incorporating retrieval augmented generation (RAG) to enhance response accuracy and reduce computational overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large LLMs are used to improve response quality and capabilities, then the model can outperform smaller models on complex tasks, but computational burden and inference time increase significantly
Solution Approach 1:
The patent divides the large LLM into multiple smaller specialized models, each trained on specific datasets for particular tasks. Instead of using one large model for all tasks, the system segments functionality across multiple smaller models that can be invoked selectively based on the input query, reducing overall inference time while maintaining response quality.
Solution Approach 2:
The system performs preliminary classification of input queries to determine which specialized smaller model should handle each task. By pre-categorizing queries and routing them to appropriate specialized models before inference, the system avoids the overhead of running a large general-purpose model for every query, thus reducing inference time while maintaining high response quality for task-specific queries.
2Adaptability or versatility
If large LLMs are deployed to handle complex tasks, then model capabilities improve, but computational resources and infrastructure requirements increase
Solution Approach 1:
The patent segments the monolithic large LLM architecture into multiple smaller specialized models, each handling specific task types. This segmentation reduces the computational resources required for each individual model while collectively providing diverse capabilities across the ensemble, lowering infrastructure requirements compared to a single large model.
Solution Approach 2:
The system creates a universal framework that can handle multiple task types through a collection of specialized models. Each smaller model is trained for specific functions but the overall system provides multi-functionality by routing different queries to appropriate models, achieving versatility without requiring each model to be large and resource-intensive.
3Measurement precision
If larger LLMs are trained to improve performance, then model accuracy increases, but training time and computational cost increase
Solution Approach 1:
The patent segments the training process into multiple specialized training campaigns, each focused on a specific task or domain. Instead of training one large model on all possible tasks (which would require extensive time and resources), the system trains multiple smaller models on specialized datasets, achieving high accuracy for each task domain more efficiently.
Solution Approach 2:
The system changes the parameters of the models by using different sizes and architectures optimized for specific tasks rather than uniformly large models. Each specialized model uses parameters and architecture appropriate for its specific function, reducing overall training time while maintaining or improving accuracy for task-specific queries compared to a general large model.
4Productivity
If specialized smaller models are used to reduce computational burden, then inference time decreases, but model versatility and task coverage may be limited
Solution Approach 1:
The patent implements a universal query routing framework that directs different types of queries to appropriate specialized models. The system maintains versatility by covering multiple task domains through a collection of specialized models, each optimized for specific functions, while the routing mechanism ensures comprehensive task coverage similar to a large general-purpose model but with faster inference speeds for task-specific queries.
Solution Approach 2:
The system performs preliminary query analysis and classification to route inputs to the most appropriate specialized model. This preliminary action enables the smaller specialized models to handle their specific tasks efficiently with fast inference, while the routing framework ensures broad task coverage by directing different query types to the models best suited for them.
Data Source
AI summary
Apparatus and methods are disclosed for implementing a copilot as a network of microservices including specialized large language models (LLMs) or other trained machine learning (ML) tools. This architecture supports flexible, customizable, or dynamically determinable dataflow. Compared to much larger competing LLMs, comparable or superior performance is achieved, while significantly reducing computation time and hardware requirements, even to a single compute node with a single GPU. Examples incorporate a retrieval microservice, as least one data producer, and a core microservice. Based on client input, the retrieval microservice can perform multiple iterations of retrieval augmented generation (RAG). At each iteration, output (based on any preceding iterations' results or the client input) is transmitted to a data producer, and results received therefrom. Eventually, based on these results, an output is transmitted toward the core microservice for generation of a response to the client input. Variations and additional techniques are disclosed.