3D Companion Robot Interaction Using Emotion-Labeled Multimodal Responses
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional companion robots integrated with large language models lack the ability to provide realistic companionship due to limited interaction and customization, making them ineffective in simulating emotional connections with users.
Innovation Solution
A system for companion robots with 3D display that includes sensors to sense user behavior and speech, a server-end host for parsing and generating emotion-labeled chat messages, and a 3D display device to output synchronized image data from multiple angles, enhancing emotional interaction through three-dimensional visuals and audio.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a conventional large language model is used in a companion robot, then the robot can understand and generate natural language texts, but it lacks real perception and understanding abilities, cannot judge the real world, and is incapable of emotions
Solution Approach 1:
The system segments the processing of user input into multiple independent modules: speech recognition module for audio input, behavior recognition module for sensor data, emotion recognition module for analyzing user emotions, and large language model module for generating responses. Each module handles a specific aspect of interaction, allowing the robot to process complex inputs comprehensively while maintaining the simplicity of natural language processing.
Solution Approach 2:
The system merges multiple input sources (speech, behavior, emotion data) into a unified processing framework. The large language model receives integrated information from speech recognition, behavior recognition, and emotion recognition modules, enabling it to generate responses that consider all aspects of user interaction rather than processing isolated inputs.
2Adaptability or versatility
If a companion robot is integrated with a display device to play images or videos of relatives, then the function of playing images or videos is added, but the companionship effect is still very limited and users cannot feel realistic companionship
Solution Approach 1:
The system transitions from traditional 2D display (images or videos) to 3D spatial display by arranging multiple display devices in three-dimensional space around the robot. This creates a immersive visual environment where relatives appear to be physically present in the space, providing realistic companionship rather than merely playing flat images or videos.
Solution Approach 2:
The display system serves multiple functions simultaneously: it displays images and videos of relatives, provides spatial audio positioning for realistic conversation, and creates an immersive three-dimensional environment. This multi-functional approach enhances companionship effect beyond the limited function of simple media playback.
3Ease of operation
If the 3D display device displays M pieces of image data with different capture angles, then the three-dimensional display effect is achieved, but the device complexity increases
Solution Approach 1:
The display system segments the three-dimensional image data into M separate pieces, each corresponding to a different capture angle. Each piece of image data is processed and displayed by dedicated pixel lines, allowing the system to handle complex 3D rendering through modular processing rather than attempting to render the entire three-dimensional scene as a single complex image.
Solution Approach 2:
The system processes and displays M pieces of image data with different capture angles, which is more detailed than traditional single-angle display. This partial processing approach (handling each angle separately) achieves comprehensive three-dimensional coverage without requiring the system to process all possible viewing angles simultaneously, balancing detail with feasibility.
Data Source
AI summary
A system for companion robot with 3D display device and a method are disclosed. In the system, a companion robot continuously senses a user behavior status and an input speech, converts the input speech to a text, transmits the text and the user behavior status to finite state machines of a server-end host to perform parsing and transition, so that the server-end host generates a chat message having an emotion label, transmits the chat message to an AI device, receives and stores the companion conversation message to a companion conversation list; the companion robot selects companion conversation message from the companion conversation list as an output message, and convert the output message to speech which is broadcasted by a speaker; while broadcasting the output message, the companion robot continuously obtains image data from a multi-source image and outputs the image data to a 3D display device for displaying.