3D Avatar Animation via Audio-Visual Stream Conversion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional electronic messaging lacks personalization, particularly in instant messaging, as generic emojis and graphics fail to replicate the intimacy and emotional nuances of in-person communication.
Innovation Solution
A method and system for creating customized animatable 3D models of virtual characters that mimic the facial expressions of users, using input from audio and visual streams to generate dynamic animations, which can be shared through electronic messages, incorporating auto-landmarking, retopology, texture transfer, and rigging to create personalized geometry and control structures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If generic emojis and graphics are used in electronic messaging, then the messaging can be transmitted efficiently, but the personalization and emotional depth are insufficient
Solution Approach 1:
The patent creates a copy of the user's physical appearance and expressions through a 3D avatar model. The avatar replicates facial features, body type, and real-time expressions based on captured video and audio data, allowing the user to be represented accurately in digital communication without requiring complex real-time rendering systems.
Solution Approach 2:
The patent implements dynamic animation of the 3D avatar by capturing real-time user movements through video and audio streams. The system processes this data to generate animated sequences that mimic the user's gestures, facial expressions, and speech patterns, enabling the avatar to adapt to different communication scenarios automatically.
2Adaptability or versatility
If customized 3D avatars with real-time animation are implemented, then personalization and emotional expression are improved, but processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary actions by capturing video and audio data during designated periods before the actual messaging occurs. The system processes this pre-captured data to generate and store animated sequences that can be quickly retrieved and displayed when needed, avoiding real-time processing delays during communication.
Solution Approach 2:
The system maintains continuous capture and processing of user data during communication sessions, building up a library of animated sequences that can be continuously added to and retrieved from storage. This continuous operation allows for rapid retrieval of appropriate avatar animations without interrupting the communication flow.
Data Source
AI summary
Dynamically customized animatable 3D models of virtual characters (“avatars”) in electronic messaging are provided. Users of instant messaging are represented dynamically by customized animatable 3D models of a corresponding virtual character. An example method comprises receiving input from a mobile device user, the input being an audio stream and/or an image/video stream; and based on an animatable 3D model and the streams, automatically generating a dynamically customized animatable 3D model corresponding to the user, including performing dynamic conversion of the input into an expression stream and corresponding time information. The example method includes generating a link to the expression stream and corresponding time information, for transmission in an instant message, and causing display of the customized animatable 3D model. Link generation and causing display is performed automatically or in response to user action. The animatable 3D model can be customized in the cloud or downloaded for customization.