Natural-Language Sensor Fusion for Adaptive Vehicle HMI
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing HMI systems in vehicles lack flexibility and efficiency in adapting to user needs, often requiring complex engineering efforts and computational resources, and may cause user distraction due to non-natural interaction methods.
Innovation Solution
A multi-modal sensor fusion approach using a transformer-based user state estimation module and a reward model to generate natural language descriptions of user states, enabling an HMI agent to predict and adapt HMI device interactions based on user actions, reducing engineering complexity and computational load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional multi-modal sensor fusion methods are used, then user state monitoring capability is achieved, but engineering complexity and computational resources increase
Solution Approach 1:
The patent introduces natural language as an intermediary representation layer between multi-modal sensor data and user state monitoring. Instead of directly fusing complex sensor data from multiple modalities, the system converts sensor data into natural language descriptions, which then serve as the basis for determining user states. This intermediary approach simplifies the engineering complexity while maintaining monitoring capability.
2Adaptability or versatility
If complex interaction methods are used to adapt HMI systems to user needs, then adaptation accuracy improves, but user distraction increases
Solution Approach 1:
The patent replaces complex mechanical interaction methods with natural language processing. Instead of requiring users to interact with complex HMI interfaces to adapt the system to their needs, the system uses natural language descriptions of user states (derived from sensor data) to automatically adapt. This substitution reduces user distraction while maintaining or improving adaptation accuracy.
3Ease of operation
If natural language-based user state estimation is implemented, then interaction naturalness improves, but computational efficiency may decrease
Solution Approach 1:
The patent segments the computational process into distinct stages: sensor data processing, natural language generation, and user state determination. By dividing the task and using pre-trained language models for the natural language generation step, the system achieves computational efficiency while maintaining interaction naturalness. The segmentation allows each component to be optimized independently.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The invention proposes a computer-implemented method for operating an HMI system (16), the method comprising: a) monitoring a user with a multi-modal sensor arrangement (20) to obtain multi-modal sensor data (30) that are indicative of a user state; b) feeding the multi-modal sensor data (30) to a user state estimation module (32), wherein the user state estimation module (32) processes the multi-modal sensor data (30) and outputs a user state embedding (34) that represents a natural language description of the user state or the natural language description of the user state; c) feeding the result of step b) to an HMI agent (40) and a reward model (36), wherein the reward model (36) determines a reward value (38) and feeds it to the HMI agent (40), and the HMI agent (40) determines a control signal (42) that is configured to cause a modification to an HMI device (18).