Quantized Foundation Model Pipeline for Real-Time Embedded Devices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial intelligence technologies face challenges in providing real-time and efficient generative intelligence on embedded devices due to resource constraints such as power consumption, size, and performance limitations.
Innovation Solution
A foundation model pipeline is developed for real-time embedded devices, utilizing agentic prompt generation and adaptive prompt augmentation techniques to enhance the functionality of large language models (LLMs) and large multi-modal models (LMMs) on resource-constrained devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional large language models are deployed on embedded devices, then generative intelligence functionality is provided, but resource consumption exceeds device capabilities
Solution Approach 1:
The patent segments the LLM processing into two distinct parts: a compressed quantized version running locally on the embedded device for real-time operations, and a full-precision version running on a cloud server for complex tasks. This segmentation allows the device to maintain generative intelligence functionality while staying within power consumption constraints by only executing lightweight operations locally.
Solution Approach 2:
The patent introduces a cloud server as an intermediary that handles the computationally intensive tasks. The embedded device sends complex prompts to the cloud server, receives processed results, and presents them to the user. This intermediary absorbs the resource consumption burden, allowing the embedded device to provide generative intelligence functionality without exceeding its power consumption limits.
2Reliability
If large language models with massive parameters are used, then complex linguistic patterns are captured, but device size and performance constraints are violated
Solution Approach 1:
The patent creates a compressed quantized copy of the large language model that retains the essential linguistic patterns and generation capabilities while significantly reducing the number of parameters. This compressed copy can run on the embedded device and produce coherent text, while the full-precision model remains on the cloud server for reference and complex processing.
Solution Approach 2:
The patent applies partial action by using a quantized version of the model with reduced parameters for local execution. Instead of loading the complete massive parameter model onto the device, only the necessary compressed version is deployed, achieving sufficient text generation reliability while respecting device complexity constraints.
3Speed
If real-time processing is required, then timely responses are achieved, but processing power consumption increases
Solution Approach 1:
The patent segments processing tasks by complexity: simple real-time prompts are handled by the compressed local model for fast response, while complex prompts are offloaded to the cloud server. This segmentation enables real-time processing for common tasks without requiring the device to continuously consume high processing power for all operations.
Solution Approach 2:
The compressed quantized model on the embedded device handles routine real-time generation tasks independently, serving the user's immediate needs without requiring continuous connection to or power consumption from the cloud server. This self-service capability provides real-time responses for common queries while minimizing overall power consumption.
Data Source
AI summary
Systems, computer programs, devices, and methods that enable LLM-based user interfaces within real-time and/or embedded devices. Providing user-specific context to a generically trained LLM may enable a variety of new usages and scenarios. For example, adaptive prompt augmentation may enable a user device to augment user-generated prompts with additional user context in the form of machine-generated prompts. In some variants, machine-generated prompts may be further refined to accommodate e.g., foundation model constraints, etc. APIs for user-specific data structures can be used to e.g., optimize for habitual behaviors, user idiosyncrasies, etc. Agentic query construction may enable a user device to operate with autonomy and decision-making capabilities, beyond prompt-response interactions. Stitching (or dreaming) may be used to identify pattern-based associations within high dimensional space (embedding vectors).