Robot Conversation Generation Using Multimodal Emotion Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing LLM-based conversation systems struggle to reflect emotional states and non-verbal expressions, leading to uncomfortable interactions and reduced usability, and require high calculation resources, resulting in latency issues during real-time conversations.
Innovation Solution
A method utilizing multi-modal sensing technology to collect and analyze speech and facial expressions, combined with fine-tuning techniques and semantic communication, to generate emotional responses efficiently and in real-time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If an LLM-based conversation system processes only text input, then the calculation resource requirement is reduced, but the system cannot reflect emotional states or non-verbal expressions
Solution Approach 1:
The system segments the conversation processing into two parts: (1) multi-modal emotion recognition module that processes speech and image inputs to extract emotional features, and (2) LLM module that processes only the extracted emotional features and text. This segmentation allows the system to incorporate emotional information without requiring the LLM to process raw multi-modal data, thus reducing calculation resource requirements while improving adaptability.
Solution Approach 2:
The patent introduces an intermediary emotion recognition module that acts as a bridge between multi-modal inputs (speech and image) and the LLM. This intermediary extracts and encodes emotional features into a compact representation that the LLM can efficiently process, thereby enabling emotional adaptability without proportionally increasing calculation resources.
2Measurement precision
If the system transmits full conversation content to the LLM server for processing, then the response accuracy is improved, but the response speed decreases due to high calculation resources and network latency
Solution Approach 1:
The system extracts only the essential emotional features and key contextual information from the full conversation content, rather than transmitting all raw data to the LLM server. This extraction process removes unnecessary data while retaining the critical information needed for accurate emotional response generation, thereby improving response speed without significantly compromising accuracy.
Solution Approach 2:
The patent transforms the conversation content from raw multi-modal data into encoded emotional feature vectors and condensed contextual representations. This parameter transformation reduces the dimensionality and complexity of the data transmitted to the server, enabling faster processing while maintaining the essential information needed for accurate responses.
3Measurement precision
If the system uses multi-modal sensing to collect speech and facial expressions, then the emotional analysis precision is improved, but the device complexity increases
Solution Approach 1:
The system employs a unified multi-modal emotion recognition module that simultaneously processes both speech and facial expression inputs through integrated neural network architectures. This universal module performs multiple functions (speech recognition, facial expression analysis, emotion classification) within a single framework, thereby improving emotional analysis precision while managing device complexity through functional integration rather than separate dedicated components for each modality.
Data Source
AI summary
A conversation generation system of a robot according to one embodiment may perform operations of acquiring, by the robot, the speech and image of an uttering user; encoding, by the robot, the speech and the image to generate emotional semantic information, and encoding the speech to generate contextual semantic information to transmit the contextual semantic information and the emotional semantic information to the LLM server; decoding, by the LLM server, the emotional semantic information to extract the user's emotional information derivable from the speech and the image, decoding the contextual semantic information to extract contextual information including text included in the speech, and applying a RAG technique to the emotional information and the contextual information to generate augmented information; inputting, by the LLM server, the emotional information, the contextual information, and the augmentation information into an LLM model to derive a response of the LLM model to the contextual information reflecting the emotional information; encoding, by the LLM server, the response to generate response semantic information and transmit it to the robot; and decoding, by the robot, the response semantic information to output a response to the user's utterance.