Cloud-Assisted In-Vehicle LLM Accuracy via Logit Adjustment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Vehicle-installed Large Language Models (LLMs) face accuracy limitations due to storage and processing constraints, and using cloud-based LLMs risks personal information leakage, which can degrade output accuracy when personal information is removed.
Innovation Solution
A system that uses a small LLM in the vehicle and a larger LLM in the cloud, where personal information is redacted from the input prompt, and the difference in logits between smaller and larger LLMs is applied to enhance output accuracy without sending personal information to the cloud, utilizing identical parameter sets with different values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a larger LLM is used in the vehicle to improve output accuracy, then output accuracy is improved, but storage and processing constraints are exceeded
Solution Approach 1:
The system segments the LLM functionality into two parts: a smaller LLM deployed in the vehicle for basic processing, and a larger LLM deployed in the cloud for enhanced processing. This segmentation allows the vehicle to maintain LLM functionality while offloading the computational burden of running a large model to the cloud infrastructure.
Solution Approach 2:
The system introduces an intermediary mechanism (logit comparison and adjustment process) that bridges the smaller local LLM and the larger cloud-based LLM. By comparing logits and applying adjustments, the system enables the smaller model to achieve output quality closer to the larger model without directly running the large model in the vehicle.
2Measurement precision
If personal information is sent to the cloud LLM to improve output accuracy, then output accuracy is improved, but personal information leakage risk increases
Solution Approach 1:
The system extracts and removes personal information from the input prompt before sending it to the cloud-based LLM. By taking out the harmful element (personal information) while retaining the essential context needed for accurate processing, the system enables cloud processing without compromising user privacy.
Solution Approach 2:
The system creates a modified copy of the input prompt with personal information removed or anonymized before transmitting to the cloud. This copying approach allows the cloud LLM to process the request accurately while working with a safe, de-identified version of the original input.
3Object-affected harmful factors
If personal information is redacted from the input prompt to protect privacy, then personal information leakage risk is reduced, but output accuracy deteriorates
Solution Approach 1:
The system implements a feedback mechanism where the smaller local LLM's output is compared against the larger cloud-based LLM's output. The logit differences serve as feedback signals that guide adjustments to improve the local model's accuracy, allowing the system to compensate for the removed personal information through iterative refinement.
Solution Approach 2:
The system changes the parameters of the local LLM by applying logit adjustments derived from comparing its output with the cloud-based LLM. This parameter modification allows the local model to adapt its behavior and achieve higher accuracy despite operating on redacted input without personal information.
4Productivity
If a smaller LLM is used in the vehicle to reduce computational load, then computational resources are reduced, but output accuracy deteriorates
Solution Approach 1:
The system transitions from a single-dimension approach (running one LLM size) to a multi-dimensional approach by utilizing both a smaller local LLM and a larger cloud-based LLM. This dimensional expansion allows the system to leverage the speed of the small model and the accuracy of the large model through logit comparison and adjustment.
Solution Approach 2:
The system creates a composite intelligence architecture that combines the strengths of two different LLM sizes. By integrating the outputs of a smaller local model and a larger cloud-based model through logit manipulation, the system achieves a composite solution that balances computational efficiency with high output accuracy.
Data Source
AI summary
Cloud-assisted in-vehicle large language model use is performed by inputting an unredacted prompt into a first Large Language Model ( LLM ) installed in a vehicle, wherein the unredacted prompt includes personal information relating to an occupant of the vehicle, transmitting a redacted prompt to a second LLM installed in a cloud, the redacted prompt representing the unredacted prompt without the personal information, receiving a logit of the second LLM, inputting the redacted prompt into a third LLM installed in the cloud or the vehicle, the third LLM including an identical parameter set to the first LLM, determining a difference between a logit of the first LLM and a logit of the third LLM, determining a sum of the difference and the logit of the second LLM to obtain an adjusted logit, and acquiring an output from the adjusted logit.