Foundation Model Pipeline for Real-Time Embedded LLMs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial intelligence technologies face challenges in providing real-time and efficient generative intelligence capabilities on embedded devices, which are often constrained by limited resources such as power consumption, size, and performance.
Innovation Solution
The implementation of a foundation model pipeline that includes agentic prompt generation and adaptive prompt augmentation, utilizing user devices like smart glasses and smartphones to enhance the functionality of large language models (LLMs) on embedded devices, allowing for real-time interactions and efficient resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional large language models are deployed on embedded devices, then generative intelligence functionality is provided, but device resources (power consumption, size, performance) are exceeded
Solution Approach 1:
The system segments the LLM processing into multiple components: a lightweight local model running on the embedded device for immediate responses, and a cloud-based foundation model for complex tasks. This segmentation allows the device to provide generative intelligence functionality while consuming less power by only executing the lightweight local model.
Solution Approach 2:
The patent introduces an intermediary cloud service that handles computationally intensive foundation model processing. The embedded device communicates with the cloud service to offload heavy computations, thereby reducing local power consumption while maintaining access to powerful generative intelligence capabilities.
2Adaptability or versatility
If conventional large language models are deployed on embedded devices, then generative intelligence functionality is provided, but device size and performance constraints are violated
Solution Approach 1:
The system divides the AI functionality into a lightweight local component and a cloud-based foundation model component. The embedded device only needs to host the lightweight model and communication interfaces, significantly reducing device complexity and resource requirements compared to running full LLMs locally.
Solution Approach 2:
The patent implements a universal architecture where the same lightweight local model can serve multiple functions: running independently for simple tasks, collaborating with cloud services for complex tasks, and adapting to different applications. This multi-functionality provides generative intelligence capabilities without requiring device-specific heavy resources.
3Productivity
If real-time processing is required for user interactions, then responsiveness is improved, but processing time constraints are difficult to meet with conventional LLMs
Solution Approach 1:
The system segments processing tasks by complexity: simple queries are handled by the lightweight local model with fast response times, while complex queries are delegated to the cloud foundation model. This segmentation ensures real-time responsiveness for common interactions while maintaining access to sophisticated models for complex tasks.
Solution Approach 2:
The patent applies partial action by using the lightweight local model to handle only the portion of processing that is immediately necessary for real-time responses, while deferring the excessive computational action of full foundation model processing to appropriate times when cloud resources are available.
4Adaptability or versatility
If foundation model capabilities are made accessible on embedded devices, then adaptability is improved, but resource consumption increases
Solution Approach 1:
The cloud service acts as an intermediary that provides foundation model capabilities to embedded devices without requiring the devices to host the full models. This intermediary approach enables access to sophisticated capabilities while minimizing local energy consumption through efficient communication and computation offloading.
Data Source
AI summary
Systems, computer programs, devices, and methods that enable LLM-based user interfaces within real-time and/or embedded devices. Providing user-specific context to a generically trained LLM may enable a variety of new usages and scenarios. For example, adaptive prompt augmentation may enable a user device to augment user-generated prompts with additional user context in the form of machine-generated prompts. In some variants, machine-generated prompts may be further refined to accommodate e.g., foundation model constraints, etc. APIs for user-specific data structures can be used to e.g., optimize for habitual behaviors, user idiosyncrasies, etc. Agentic query construction may enable a user device to operate with autonomy and decision-making capabilities, beyond prompt-response interactions. Stitching (or dreaming) may be used to identify pattern-based associations within high dimensional space (embedding vectors).