Hybrid Generative AI Models for Edge Resource Constraints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generative artificial intelligence models are computationally expensive and impractical for deployment on devices with limited resources due to their large size and computational requirements, making it challenging to generate accurate responses across various devices with different capabilities.
Innovation Solution
A hybrid computing environment is implemented, where differently sized generative models are trained and deployed based on device capabilities, with queries routed to appropriate devices for processing, and contextual information from external knowledge repositories is used to enhance response accuracy, allowing for efficient resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a large language model is used to generate each token in a response, then response generation accuracy is improved, but computational expense increases significantly
Solution Approach 1:
The system segments the response generation process into two distinct phases: a drafting phase using a smaller, faster language model to generate initial tokens, and a verification phase using a larger, more accurate language model to review and correct only the generated tokens. This segmentation allows the system to achieve high accuracy without the computational cost of using only the large model for all generation operations.
Solution Approach 2:
The system applies partial action by having the large language model perform verification only on the tokens generated by the small model, rather than generating all tokens itself. The large model's full computational power is not continuously deployed, but rather applied selectively to verify and correct the draft tokens, reducing overall computational expense while maintaining accuracy.
2Measurement precision
If a large language model is deployed on edge devices, then response quality is improved, but device resource constraints are exceeded
Solution Approach 1:
The system segments the language model functionality into two separate models with different sizes and capabilities. The smaller model is deployed on edge devices for initial token generation, while the larger model is deployed in the cloud for verification. This segmentation enables edge devices with limited resources to participate in the system without needing to host the computationally intensive large language model.
Solution Approach 2:
The smaller language model acts as an intermediary between the user's edge device and the large language model in the cloud. It handles the initial generation task locally, reducing the computational burden on edge devices, while still benefiting from the large model's accuracy through the verification phase. This intermediary approach allows edge devices to access high-quality responses without directly hosting the resource-intensive large model.
3Measurement precision
If contextual information is retrieved from external repositories, then response accuracy is improved, but processing time increases
Solution Approach 1:
The system performs preliminary action by retrieving and processing contextual information from external knowledge repositories during the drafting phase, before the verification phase begins. This allows the contextual data to be ready and integrated into the draft response, so that when the large model performs verification, it can focus solely on accuracy refinement without additional time for data retrieval.
Data Source
AI summary
Certain aspects of the present disclosure provide techniques and apparatus for generating a response to an input query using a generative artificial intelligence model. An example method generally includes receiving an input for processing. A prompt representing the received input is generated based on the received input, contextual information associated with the received prompt, and a prompt-generating artificial intelligence model. The generated prompt is output to a generative artificial intelligence model for processing. A response to the generated prompt is received from the generative artificial intelligence model and output as a response to the received input.