Multimodal Embeddings for Edge-Cloud Assistant Privacy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automated assistants do not effectively incorporate visual data into ongoing conversations, leading to inefficiencies and privacy concerns when processing sensor data from multiple modalities.
Innovation Solution
Implementing a local, scaled-down multimodal LLM on a device to encode various sensor data modalities into embeddings, which are then transmitted to a cloud-based LLM for processing, preserving privacy and conserving computing resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If visual data is incorporated into ongoing conversations with automated assistants, then multimodal engagement is enhanced, but network bandwidth and computing resources are consumed
Solution Approach 1:
The system segments the processing architecture into edge-based components (encoders and LLMs deployed on client devices) and cloud-based components (central LLM on remote server). This segmentation allows local processing of visual and audio data into embeddings, reducing the amount of data transmitted over the network while maintaining multimodal processing capabilities.
Solution Approach 2:
The patent introduces multimodal embeddings as an intermediary representation that bridges visual and audio modalities. These embeddings serve as compressed, semantically rich representations that can be processed by the central LLM without transmitting the original high-bandwidth sensory data, thus conserving network resources while enabling multimodal conversation.
2Measurement precision
If raw sensor data is transmitted to the cloud for processing, then processing accuracy is improved, but user privacy is compromised and network bandwidth is consumed
Solution Approach 1:
The system extracts only the essential semantic information from raw sensor data by converting visual and audio inputs into multimodal embeddings at the edge device. This extraction process removes personally identifiable and sensitive information while retaining the semantic meaning needed for accurate processing by the central LLM.
Solution Approach 2:
The patent transforms raw sensor data from its original high-dimensional form into compressed embedding representations with reduced dimensionality. This parameter transformation maintains the essential semantic properties needed for accurate processing while reducing the amount of data transmitted and stored, thereby protecting user privacy.
Data Source
AI summary
Implementations relate to generating and using multimodal embeddings. In various implementations, first modality data may be obtained and encoded into first modality embedding(s) using a trained first modality encoder that is stored in memory of edge-based client device(s). Second modality data may be obtained and encoded into second modality embedding(s) using a trained second modality encoder that is also stored in the memory of the edge-based client device(s). The first and second modality embeddings may be processed using an edge-based multimodal LLM that is also stored locally in memory of the edge-based client device(s) to generate a multimodal contextual embedding, which may be provided to a remote server that hosts a central LLM, e.g., in conjunction with a natural language input provided by the user. Information generated using the central LLM, responsive to the natural language input, may be received from the remote server.