Segmented Foundation Model Pipelines for Real-Time Embedded Devices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional large language models (LLMs) are resource-intensive and lack the ability to operate in real-time or embedded devices due to computational and power constraints, requiring modification to access contextual data and leading to unstable user experiences.
Innovation Solution
Implementing agentic prompt generation through user device augmentation, including machine-generated prompt augmentation and persona-based stitching, to enhance LLM functionality in real-time embedded devices by leveraging on-device data and user-specific information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional large language models are deployed in real-time embedded devices, then generative intelligence functionality is provided, but computational resources and power consumption exceed device constraints
Solution Approach 1:
The system segments the LLM pipeline into distinct components: a reduced complexity foundation model running on the embedded device for local processing, and a cloud-based enhanced model for complex tasks. This segmentation allows the device to handle simple generative tasks locally without consuming excessive power, while escalating to cloud resources only when needed for more demanding computations.
Solution Approach 2:
A cloud service acts as an intermediary between the embedded device and full LLM functionality. The cloud service receives simplified prompts from the device's reduced model, enhances them with contextual data, and returns refined responses. This intermediary approach enables the device to access powerful generative capabilities without running the full computationally intensive model locally.
2Adaptability or versatility
If conventional large language models are deployed in real-time embedded devices, then generative intelligence functionality is provided, but computational performance is insufficient for real-time operation
Solution Approach 1:
The system segments processing tasks by complexity: simple, time-sensitive generative tasks are handled by a reduced complexity foundation model running locally on the device at high speed, while complex tasks requiring extensive computation are offloaded to the cloud. This segmentation enables real-time response for common operations without being bottlenecked by the computational requirements of full LLMs.
Solution Approach 2:
The system uses a simplified, reduced complexity foundation model as a disposable proxy for the full LLM. This lightweight model provides sufficient generative intelligence for real-time embedded applications without the computational burden of the complete model, effectively replacing the need to run the full heavy model continuously.
3Adaptability or versatility
If conventional large language models are deployed in embedded devices, then generative intelligence is provided, but the models cannot access contextual data from diverse sources
Solution Approach 1:
The cloud service functions as an intermediary that bridges the embedded device with diverse data sources. It receives simplified prompts from the device, enriches them with contextual information from various sources, and returns enhanced responses. This intermediary layer provides sophisticated contextual data access capabilities without requiring the device to maintain complex direct connections to multiple data sources.
Solution Approach 2:
The system extracts contextual data access capabilities from the device and places them in the cloud service. The embedded device simply sends simplified prompts to the cloud, which retrieves and processes contextual information from diverse sources. This extraction approach provides rich contextual data functionality while keeping the device architecture simple.
4Adaptability or versatility
If conventional large language models are deployed in embedded devices, then generative intelligence is provided, but user experience becomes unstable due to resource constraints
Solution Approach 1:
The system dynamically adjusts the processing approach based on task complexity and device resources. Simple tasks are handled locally by the reduced model for fast, stable responses, while complex tasks are offloaded to the cloud. This dynamic adaptation ensures consistent, stable user experience by matching the computational approach to the specific demands of each interaction, preventing resource exhaustion and performance degradation.
Data Source
AI summary
Systems, computer programs, devices, and methods that enable LLM-based user interfaces within real-time and/or embedded devices. Providing user-specific context to a generically trained LLM may enable a variety of new usages and scenarios. For example, adaptive prompt augmentation may enable a user device to augment user-generated prompts with additional user context in the form of machine-generated prompts. In some variants, machine-generated prompts may be further refined to accommodate e.g., foundation model constraints, etc. APIs for user-specific data structures can be used to e.g., optimize for habitual behaviors, user idiosyncrasies, etc. Agentic query construction may enable a user device to operate with autonomy and decision-making capabilities, beyond prompt-response interactions. Stitching (or dreaming) may be used to identify pattern-based associations within high dimensional space (embedding vectors).