Semantic Query Matching for Copilot API Data Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large Language Models (LLMs) face challenges with high computational burdens, latency, and undesirable artifacts such as inaccuracies, biases, and hallucinations, limiting their practical application in various domains.
Innovation Solution
A microservice architecture utilizing small to mid-sized trained machine learning tools, each with specialized functions, is employed to handle client inputs efficiently, allowing for customized deployment, reduced computational effort, and enhanced safety through dynamic dataflow and domain-specific training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If large LLMs are used to improve language processing capabilities, then accuracy and reasoning performance are improved, but computational burden and latency increase
Solution Approach 1:
The patent divides the monolithic large LLM into multiple specialized microservices, each handling specific tasks (translation, summarization, Q&A, code generation). This segmentation allows parallel processing of different aspects of language understanding, reducing overall inference latency while maintaining comprehensive accuracy through coordinated specialized models.
Solution Approach 2:
The patent creates a universal language processing system where a single LLM is trained to perform multiple functions (translation, summarization, Q&A, code generation) simultaneously. This multi-functional approach eliminates the need for separate specialized models for each task, reducing computational overhead while maintaining versatility across different language processing needs.
2Measurement precision
If large LLMs are used to improve language processing capabilities, then accuracy and reasoning performance are improved, but computational resources required increase
Solution Approach 1:
The patent segments the computational workload across multiple smaller specialized models rather than using one large model. Each microservice runs a smaller LLM optimized for its specific task, reducing the computational resources required for each individual processing unit while maintaining overall system accuracy through the collective expertise of specialized models.
Solution Approach 2:
The patent changes the parameters of the language models by training smaller LLMs with task-specific datasets rather than using a single large model with billions of parameters. This parameter optimization allows each specialized model to achieve high accuracy for its specific function with significantly reduced computational resource requirements compared to a general-purpose large LLM.
3Measurement precision
If large LLMs are used to improve language processing capabilities, then comprehensive knowledge is improved, but artifact generation (inaccuracies, biases, hallucinations) increases
Solution Approach 1:
The patent segments the knowledge processing into specialized domains, with each microservice handling specific types of content (translation, summarization, Q&A, code generation). This domain-specific segmentation reduces the likelihood of artifacts because each model focuses on a narrow knowledge area where it can achieve high accuracy without generating hallucinations or biases from unrelated domains.
Solution Approach 2:
The patent implements feedback mechanisms where the LLM's outputs are evaluated and refined through multiple processing stages. The system uses self-correction and verification processes where generated content is checked for accuracy and consistency before final output, reducing artifact generation while maintaining comprehensive knowledge coverage through iterative refinement.
Data Source
AI summary
Apparatus and methods are disclosed for retrieving data, responsive to received input, at a microservice supporting an application programming interface (API). A score is generated, indicating likeness of semantic content, between each of multiple candidate API-conforming queries and the received input. Queries are selected based on their scores, and executed on a live repository. Based on retrieved data, a response to the input is formulated and transmitted. Disclosed techniques are suitable for a data producer front end in a copilot having a microservice network architecture. Compared to much larger competing LLMs, comparable or superior performance is achieved for certain tasks, while computation time and hardware requirements are significantly reduced, even to a single compute node with a single GPU. One or more data producers can provide a retrieval microservice with access to various databases having respective APIs, to extend the copilot's reach. Variations and additional techniques are disclosed.