Conversational AI Gesture Prompting for Multimodal Interaction

The system segments the gesture generation process into distinct modules: an NLU model processes user queries to extract intent and entities, a gesture classifier categorizes the intent into gesture types (deictic, iconic, metaphoric, beat), and a gesture prompt generator creates specific gestural responses. This modular segmentation enables complex multimodal interaction while maintaining manageable system complexity through divided functionality.

The conversational AI system is enhanced with multi-functionality by integrating both language-based and gesture-based communication capabilities within a single unified system. The same NLU infrastructure that handles speech recognition and natural language processing is extended to also drive gesture generation, allowing the system to perform multiple communication functions (verbal and non-verbal) through a common architectural base.

The system performs preliminary action by pre-training the NLU model and gesture classifier on comprehensive datasets that include gesture annotations. During runtime, the pre-trained models quickly process user queries and generate appropriate gestures without requiring intensive real-time computation. The heavy computational work is shifted to the offline training phase, reducing the computational load during actual interaction.

The system uses copying by leveraging existing NLU models and dialog management infrastructure that are already trained for language processing. Instead of building entirely new systems for gesture recognition and generation, the patent adapts and extends existing language-based NLU components to also handle gesture classification and generation, reusing proven models and reducing computational overhead.