Agentic Foundation Model Pipelines for Real-Time Embedded Devices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional large language models (LLMs) are resource-intensive and lack the ability to operate independently, making them unsuitable for real-time and embedded applications due to their reliance on user input and inefficient resource utilization, leading to unstable user experiences.
Innovation Solution
Implementing agentic prompt generation through user device augmentation, which includes machine-generated prompt augmentation and persona-based stitching to enhance user prompts with contextual and personal information, allowing LLMs to operate independently and efficiently in real-time environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional large language models are used, then language generation capability is improved, but resource consumption increases and real-time operation becomes impossible
Solution Approach 1:
The system segments the LLM functionality into two parts: a lightweight local model that handles basic language generation tasks independently, and a cloud-based enhanced model that provides advanced capabilities only when needed. This segmentation allows the embedded device to operate autonomously with minimal resource consumption while accessing powerful language generation capabilities through cloud API calls only for complex tasks.
Solution Approach 2:
The patent introduces an intermediary layer consisting of a lightweight local LLM that acts as a mediator between user inputs and the powerful cloud-based LLM. This intermediary handles basic processing locally, filtering and preparing inputs for cloud processing only when necessary, thereby reducing overall resource consumption while maintaining language generation capability.
2Adaptability or versatility
If conventional large language models are used, then language generation capability is improved, but device complexity increases and embedded deployment becomes difficult
Solution Approach 1:
The patent extracts the complex LLM processing functionality from the embedded device and places it in the cloud. Only a simplified client interface and lightweight local model remain on the device, dramatically reducing device complexity while maintaining language generation capability through remote cloud processing.
Solution Approach 2:
The system uses a universal cloud-based LLM service that can handle multiple language generation tasks and scenarios through a single API interface. This multi-functional cloud service replaces the need for multiple specialized models on the device, simplifying the embedded system architecture while providing versatile language generation capabilities.
3Adaptability or versatility
If conventional large language models are used, then language generation capability is improved, but response time increases and real-time operation fails
Solution Approach 1:
The system performs preliminary action by having a lightweight local LLM ready to immediately process and respond to simple user inputs without waiting for cloud processing. This preliminary local processing capability ensures fast real-time responses for basic tasks, while complex tasks are asynchronously processed in the cloud without blocking the user experience.
Solution Approach 2:
The patent applies partial action by using a lightweight local LLM that performs only the necessary basic language processing locally, rather than running the full powerful cloud model continuously. This partial execution of LLM functionality locally provides fast real-time responses for simple tasks while avoiding the resource intensity and latency of full model execution.
4Adaptability or versatility
If conventional large language models are used, then language generation capability is improved, but reliability decreases due to dependency on user input and external resources
Solution Approach 1:
The system implements self-service by enabling the embedded device to autonomously process language inputs using the lightweight local LLM without requiring continuous external cloud resources or complex user input configurations. The device can independently handle basic language generation tasks, improving reliability by eliminating dependencies on external services for routine operations.
Solution Approach 2:
The lightweight local LLM performs preliminary processing and decision-making locally before involving cloud resources. This preliminary action ensures that the system can operate reliably and autonomously for basic tasks without external dependencies, while still having access to enhanced cloud capabilities when needed for complex scenarios.
Data Source
AI summary
Systems, computer programs, devices, and methods that enable LLM-based user interfaces within real-time and/or embedded devices. Providing user-specific context to a generically trained LLM may enable a variety of new usages and scenarios. For example, adaptive prompt augmentation may enable a user device to augment user-generated prompts with additional user context in the form of machine-generated prompts. In some variants, machine-generated prompts may be further refined to accommodate e.g., foundation model constraints, etc. APIs for user-specific data structures can be used to e.g., optimize for habitual behaviors, user idiosyncrasies, etc. Agentic query construction may enable a user device to operate with autonomy and decision-making capabilities, beyond prompt-response interactions. Stitching (or dreaming) may be used to identify pattern-based associations within high dimensional space (embedding vectors).