Robot Interaction Behavior Using Pose Estimation and Gesture Priority
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing social robots struggle to respond effectively in complex communicative contexts where both utterance and nonverbal communication are simultaneously performed, leading to uniform and inadequate responses due to their inability to consider multi-factor communicative situations.
Innovation Solution
A method and apparatus for generating robot interaction behavior using a pre-trained neural network model for robot pose estimation, which estimates next joint positions based on user and robot joint positions, and modifies nonverbal behavior based on emotional states, allowing for the generation of co-speech gestures and nonverbal behaviors that can overlap or prioritize each other depending on contextual inputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional single-factor communication schemes are used, then the robot can perform nonverbal behavior or utterance communication, but the robot cannot respond to complex communicative situations where both are performed simultaneously
Solution Approach 1:
The communication system is segmented into separate modules: a nonverbal behavior generation unit that processes user behavior input, a co-speech gesture generation unit that processes utterance input, and a final behavior decision unit that integrates both. This modular segmentation allows each module to handle specific communication aspects independently while maintaining overall system adaptability to complex situations.
Solution Approach 2:
The system transitions from single-factor communication to multi-dimensional communication by adding a temporal dimension for behavior sequence generation and a hierarchical dimension for behavior prioritization. The final behavior decision unit operates at a higher decision level to resolve conflicts between nonverbal behaviors and co-speech gestures, enabling responses to complex communicative situations.
2Adaptability or versatility
If pre-defined behavior patterns are used, then the robot can perform standardized nonverbal behaviors, but the generated nonverbal behavior is uniform and cannot respond to a wide variety of situations
Solution Approach 1:
The behavior generation system is made dynamic through sequential behavior generation based on current state and user input. The nonverbal behavior generation unit generates behaviors dynamically according to the current communicative situation rather than selecting from fixed patterns, enabling varied and natural responses to different situations while maintaining behavioral coherence through state transitions.
3Adaptability or versatility
If both co-speech gesture and nonverbal behavior are generated simultaneously, then the robot can respond to complex communicative contexts, but conflict resolution between behaviors is required
Solution Approach 1:
The final behavior decision unit performs preliminary conflict resolution by determining priorities between nonverbal behaviors and co-speech gestures before execution. The system pre-establishes priority rules that allow smooth integration of multiple communication modes without runtime conflicts, enabling the robot to handle simultaneous communication modes effectively.
Data Source
AI summary
Disclosed herein are an apparatus and method for generating robot interaction behavior. The method for generating robot interaction behavior includes generating co-speech gesture of a robot corresponding to utterance input of a user, generating a nonverbal behavior of the robot, that is a sequence of next joint positions of the robot, which are estimated from joint positions of the user and current joint positions of the robot based on a pre-trained neural network model for robot pose estimation, and generating a final behavior using at least one of the co-speech gesture and the nonverbal behavior.