Communication Robot Gaze Control with Integrated Audio-Visual Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing communication robot technologies fail to integrate both visual and auditory attention effectively, focusing only on visual clues without considering actual audio sources from a 3D environment and neglecting auditory attention, and primarily relying on top-down attention instead of bottom-up attention.
Innovation Solution
A communication robot system that integrates auditory and visual attention by generating an auditory attention map from sound collection and a visual attention map from face detection, combining these maps to control eyeball movements and motions using a motion processing portion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the robot uses only visual clues for attention detection, then the visual attention map can be generated using face detection and motion detection, but the robot cannot detect auditory attention or integrate both visual and auditory attention
Solution Approach 1:
The patent merges visual attention detection and auditory attention detection into a unified attention map. The visual information processing portion generates a visual attention map from face detection and motion detection results, while the auditory information processing portion generates an auditory attention map from voice volume recognition. These two maps are then integrated by the attention map generation portion to create a comprehensive attention map that combines both visual and auditory attention information, enabling the robot to detect attention from multiple sensory modalities simultaneously.
2Ease of operation
If the robot focuses on top-down attention control, then it can maintain current attention based on internal goals, but it cannot effectively respond to bottom-up stimuli from the environment
Solution Approach 1:
The patent implements a feedback mechanism where the integrated attention map continuously provides information about environmental stimuli to the motion processing portion. The attention map generation portion integrates both visual and auditory attention information, creating a comprehensive representation of salient environmental features. This integrated map feeds back to the motion processing portion, which uses it to automatically adjust the robot's gaze and body orientation, enabling bottom-up attention driven by environmental stimuli while maintaining operational ease.
3Device complexity
If the robot processes only simple visual saliency features, then the processing complexity remains low, but it cannot handle complex auditory-visual integration or 3D sound positioning
Solution Approach 1:
The patent segments the complex attention processing task into distinct functional modules: the visual information processing portion handles face detection and motion detection to generate visual attention maps, the auditory information processing portion handles voice volume recognition and 3D sound positioning to generate auditory attention maps, and the attention map generation portion integrates these separate maps. This segmentation allows each module to process its specific type of information with appropriate complexity while maintaining overall system manageability and achieving high measurement precision through specialized processing in each domain.
Data Source
AI summary
A communication robot includes an auditory information processing portion configured to recognize a volume of voice collected by a sound collection portion and generate an auditory attention map by projecting a sound position in a three-dimensional space onto a two-dimensional attention map in which the robot is located at a center, a visual information processing portion configured to generate a visual attention map using a face detection result obtained by detecting a face of a person from an image captured by an imaging portion and a motion detection result obtained by detecting a motion of the person, an attention map generation portion configured to generate an attention map by integrating the auditory attention map and the visual attention map, and a motion processing portion configured to control eyeball movements and motions of the communication robot using the attention map.