Enterprise Federated Language Models with Real-Time Confidence Cascading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current AI generative engines, such as ChatGPT™, Gemini™, Grok™, Perplexity™, and Claude™, rely on single Large Language Models (LLMs) that may hallucinate inaccurate information when faced with queries outside their training data, leading to inefficient resource utilization and manual verification burdens, and lack effective early detection of errors.
Innovation Solution
Implement a system that trains enterprise-specific federated language models (ELLMs) using enterprise data, categorizes queries, and performs real-time confidence evaluation and cascading between models to ensure accurate responses, incorporating enterprise-specific data and employee access levels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single LLM is used to process queries, then device complexity is reduced, but response reliability deteriorates due to hallucinations and inaccuracies
Solution Approach 1:
The patent segments the single LLM into multiple specialized LLMs, each trained on specific domains or tasks. This segmentation allows each model to excel in its specialized area, reducing hallucinations while maintaining manageable complexity through modular architecture.
Solution Approach 2:
The patent creates a composite system where multiple LLMs work together in an ensemble architecture. Different LLMs are combined based on their strengths, creating a composite response generation system that leverages diverse model capabilities to improve overall reliability.
2Reliability
If LLM training data is expanded to cover more domains, then response reliability improves, but loss of information increases due to data quality issues
Solution Approach 1:
The patent applies local quality by training different LLMs on high-quality, domain-specific data rather than using uniform low-quality data across all domains. Each model receives tailored training data with appropriate quality standards for its specific domain, preserving information integrity.
Solution Approach 2:
The patent segments the training data into distinct domain-specific datasets, each curated for high quality. This segmentation prevents the dilution of data quality that would occur if all domains were trained on a single large but heterogeneous dataset.
3Reliability
If computational resources are increased to improve response accuracy, then response reliability improves, but energy consumption increases
Solution Approach 1:
The patent implements partial action by using multiple smaller specialized LLMs instead of one large model. Each model processes only the queries relevant to its domain, reducing unnecessary computational energy consumption while maintaining high accuracy for specialized tasks.
Solution Approach 2:
The patent introduces dynamic model selection where the system automatically routes queries to the most appropriate LLM based on the query type and domain. This dynamic allocation optimizes energy usage by avoiding the execution of large models for simple queries that can be handled by smaller specialized models.
4Measurement precision
If error detection is performed after response generation, then measurement precision is maintained, but loss of time increases due to late-stage verification
Solution Approach 1:
The patent implements preliminary action by performing error detection and validation before final response generation. The ensemble of LLMs evaluates multiple potential responses and identifies errors in advance, allowing the system to select the most accurate response without time-consuming post-generation verification.
Data Source
AI summary
Systems and methods for training and using large language models to respond to queries and cascading to new models when their performance falls below a threshold during the response generation cycle are described. The methods involve receiving and analyzing a query using heuristics to determine the query's categories and a level of granularity at which its response is to be evaluated. Selecting a language model based on the query analysis and during the generation of the response, evaluating the model's performance for its ability to predict the next segment in a response. The methods score the evaluation and use the score to determine whether the language model's performance exceeds a confidence threshold. If it does not, then during the response is being generated, e.g., at a point before the response generation is completed, cascading from the model to a different model that can perform at a higher confidence level.