Copilot Microservice Architecture for Reduced Computational Burden
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large Language Models (LLMs) face challenges with computational burden, latency, and artifact generation, limiting their usability in various applications due to their increasing size and complexity.
Innovation Solution
A microservice architecture comprising small to mid-sized trained machine learning tools, each with specialized functions, is employed to form a network that can be efficiently deployed on single-node systems, allowing for customized training and deployment, reduced computational effort, and enhanced safety through dynamic dataflow and safety mechanisms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large LLMs with more parameters are used to improve performance, then language processing capability is improved, but computational burden and inference time increase
Solution Approach 1:
The patent divides the large LLM into multiple smaller specialized microservices, each handling specific language processing tasks. This segmentation allows the system to achieve comparable performance to large LLMs while significantly reducing computational burden, as each microservice processes only its specialized domain rather than general language tasks.
Solution Approach 2:
The patent creates a network of microservices that collectively provide universal language processing capabilities across multiple domains. Each microservice is specialized for a particular function or domain, but together they form a universal system that can handle diverse language processing tasks previously requiring a single large LLM.
2Reliability
If large LLMs are used to improve performance, then language processing capability is improved, but inference time and latency increase
Solution Approach 1:
By segmenting the large LLM into smaller specialized microservices, the system reduces inference time for each individual request. Each microservice can process its specialized tasks more quickly than a general-purpose large LLM, and the distributed architecture allows parallel processing of multiple microservices simultaneously.
Solution Approach 2:
The microservices are pre-trained on specific domains and tasks during the training phase, so that during inference they can immediately process their specialized tasks without requiring the extensive computation that would be needed for general language understanding. This preliminary specialization accelerates inference time.
3Reliability
If large LLMs are used to improve performance, then language processing capability is improved, but artifact generation increases
Solution Approach 1:
The patent segments the language processing system into specialized microservices, each with focused training on specific domains. This specialization reduces the tendency to generate artifacts because each microservice operates within its trained domain boundaries rather than making broad generalizations that large LLMs are prone to make.
Solution Approach 2:
The patent introduces intermediary components including a router that directs queries to appropriate microservices and evaluation microservices that assess output quality. These intermediaries act as mediators that can detect and prevent artifact generation by routing queries to the most appropriate specialized service and evaluating outputs before final delivery.
4Reliability
If large LLMs are used to improve performance, then language processing capability is improved, but deployment complexity increases
Solution Approach 1:
While segmentation into microservices can increase system complexity, the patent manages deployment complexity through standardized interfaces and a router architecture that simplifies coordination. Each microservice can be deployed independently, allowing for more flexible and manageable deployment compared to a single large LLM that requires significant computational resources on specialized hardware.
Solution Approach 2:
The patent uses smaller microservice models that can be copied and deployed across multiple instances more easily than a single large LLM. This copying approach allows for distributed deployment that reduces the computational burden on any single system while maintaining language processing capabilities through the network of copies.
Data Source
AI summary
Apparatus and methods are disclosed for implementing a copilot as a network of microservices including specialized large language models (LLMs) or other trained machine learning (ML) tools. The microservice network architecture supports flexible, customizable, or dynamically determinable dataflow. Compared to much larger competing LLMs, comparable or superior performance is achieved for certain tasks, while significantly reducing hardware requirements and computation time. Disclosed examples incorporate microservices for expansion, retrieval, embedding, and evaluation, in addition to one or more core microservices. Optionally, intermodal I/O, multiple data repositories, competency qualification, or human feedback can be supported. Multiple core microservices can support varying client authorizations or cognitive functions. The disclosed architecture supports any major LLM use case and can be deployed on a single compute node with a single GPU. Techniques are disclosed for multi-recursive retrieval, expansion ML tool training, input/output translation at data producers, and qualification of input/output data against copilot competency.