Federated Distributed RAG for Edge AI Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Discriminative AI systems struggle when training datasets cannot describe the complete scenario, while generative AI systems face issues like hallucinations and high parameter requirements.
Innovation Solution
A hybrid distributed inference system combining discriminative and generative AI models, with localized small foundation systems and centralized large foundation systems, utilizing retrieval augmented generation (RAG) to classify input data efficiently across edge sites and a centralized system, optimizing resource usage and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If generative AI systems are used to generalize to data not in training set, then adaptability is improved, but computer resources and parameters increase significantly
Solution Approach 1:
The system segments the AI architecture into multiple specialized components: discriminative AI models for classification, generative AI models for synthesis, and RAG systems for retrieval. Each component has a specific function and optimized parameter count, avoiding the need for a single large generative model to handle all tasks.
Solution Approach 2:
The hybrid system provides multi-functionality by combining discriminative and generative AI capabilities in a single architecture. The system can perform classification, generation, retrieval, and verification tasks using different components as needed, making it universally applicable to diverse AI tasks without requiring separate specialized systems.
2Device complexity
If discriminative AI systems are used with limited training datasets, then device complexity is reduced, but measurement precision and reliability deteriorate
Solution Approach 1:
The system merges discriminative AI models with RAG systems and generative AI models into a hybrid architecture. The discriminative models provide efficient classification while the RAG system retrieves relevant training data and the generative model synthesizes additional training examples, collectively improving classification accuracy without proportionally increasing complexity.
Solution Approach 2:
The RAG system acts as an intermediary between the discriminative AI model and the data sources. It retrieves relevant information from external databases and generates supplementary training data, bridging the gap between limited training datasets and the need for high classification accuracy on unseen data.
3Measurement precision
If centralized large foundation systems are used, then measurement precision is improved, but data transmission and processing time increase
Solution Approach 1:
The system segments inference tasks and distributes them across multiple specialized models located at different levels of the hierarchy. Simple classification tasks are handled by local discriminative models at edge devices, while more complex tasks are escalated to centralized generative models, optimizing the balance between speed and accuracy.
Solution Approach 2:
The RAG system performs preliminary actions by retrieving relevant data and pre-processing information before it reaches the generative AI model. This preprocessing reduces the computational burden on the centralized system and enables faster inference by providing focused, relevant input data.
4Loss of energy
If local small foundation systems are used at edge sites, then data transmission is reduced, but measurement precision may deteriorate
Solution Approach 1:
The system implements dynamic task routing where the complexity and location of AI inference tasks adapt based on available resources and task requirements. Local small foundation models handle routine tasks at the edge, while complex tasks are dynamically escalated to centralized systems, optimizing both data transmission efficiency and classification accuracy.
Solution Approach 2:
The system replaces the mechanical approach of always using large centralized models with a hybrid architecture that substitutes centralized processing with local processing when appropriate. The RAG system enhances local models by providing retrieved relevant data, enabling accurate local inference without requiring large model sizes.
Data Source
AI summary
The disclosure describes an enhanced distributed retrieval augmented generation (RAG) system. The enhanced distributed RAG system provides RAG data to a local small foundation system to form prompts. The enhanced distributed RAG system creates local query optimized representations (QORs) that are matched with queries sent to the local small foundation system to create prompts. The enhanced distributed RAG system is located on one edge site and is connected to the global federated RAG module. The global federated RAG system is connected to other enhanced distributed RAG systems in other edge sites. QORs can be distributed across the enhanced distributed RAG systems and the global federated RAG module by promoting local QORs to global QORs and vice versa.