Multimodal Human-Robot Interaction Planning with LLM Mediation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current frameworks for planning human-robot interaction in collaborative environments are limited by their reliance on speech input and lack of capability to process multimodal inputs, which restricts the robot's ability to engage in versatile social interactions.
Innovation Solution
A computer-implemented method that utilizes multimodal sensors to obtain information from a robot's environment, including text and non-text data, which is then converted into text-based formats processable by large language models (LLMs). This method enables the LLMs to perform behavior planning and generate multimodal interaction outputs for the robot.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If speech-only input is used for human-robot interaction, then the system complexity is reduced, but the robot's ability to engage in versatile social interactions is limited
Solution Approach 1:
The patent implements a multimodal input system that processes multiple types of human input (speech, gestures, gaze) through a unified framework. The behavior planner accepts various modalities and converts them into standardized internal representations, enabling the robot to engage in diverse social interactions without proportionally increasing system complexity. This multi-functionality approach allows single system components to handle multiple interaction types.
Solution Approach 2:
The patent introduces intermediate processing layers including modality-specific parsers and a unified behavior representation layer that mediates between diverse input modalities and the core planning system. These intermediaries translate gestures, gaze, and speech into standardized behavioral representations, allowing the robot to process multiple input types without requiring separate processing pipelines for each modality, thus managing complexity while enhancing versatility.
2Reliability
If manually setting up robot's interactive behavior is performed, then the interaction accuracy is improved, but the effort and time required increases significantly
Solution Approach 1:
The patent implements self-service mechanisms where the robot automatically learns interaction patterns and behavior rules through observed human-human interactions. The system captures real interaction data, processes it through the behavior planner, and automatically updates its interaction models without requiring manual programming. This allows the robot to improve interaction accuracy over time while eliminating the need for extensive manual setup and engineering effort.
Solution Approach 2:
The patent pre-processes and stores interaction patterns from human-human collaborations in a structured format that can be quickly retrieved and applied during robot-human interactions. By preparing behavioral templates and interaction rules in advance through automated learning from observed interactions, the system reduces both setup time and manual effort while maintaining high interaction accuracy during actual robot operations.
3Adaptability or versatility
If LLMs are used for behavior planning, then the robot can process complex language inputs, but the capability to process non-text modalities like gestures and gaze is lost
Solution Approach 1:
The patent segments the processing framework into distinct modality-specific modules (speech parser, gesture recognizer, gaze tracker) that each handle their specialized input type and convert it to standardized representations. These segmented modules feed into a unified LLM-based behavior planner, allowing complex language processing capabilities to be maintained while adding gesture and gaze processing through separate, manageable components rather than attempting to process all modalities in a single complex system.
Data Source
AI summary
A computer-implemented method for controlling a robot collaborating with a human in an environment of the robot comprises: obtaining, by at least one sensor, multimodal information on the environment of the robot including information on a human acting in the environment; converting, by a first converter, the obtained multimodal information into text information; estimating, by an intent estimator, an intent of the human based on the text information; determining, by a state estimator, a current state of the environment including the human based on the text information; planning, by a behavior planner, based on the current state of the environment and the estimated intent of the human, a behavior of the robot including at least one multimodal interaction output for execution by the robot, and generating control information including text information on the at least one multimodal interaction output; converting, by a second translator, the generated text information into multimodal actuator control information; and controlling at least one actuator of the robot based on the multimodal actuator control information.