Voice-Based Avatar Facial Expression Prediction Without Cameras
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge of realistically implementing avatar facial expressions in VR or AR environments without the use of cameras, as users' facial expressions cannot be easily tracked when wearing VR or AR devices.
Innovation Solution
A method utilizing a neural network to process voice signals, where chunks of voice data are processed through downsampling and upsampling operations to generate facial expression coefficients, enabling the prediction of avatar facial expressions based on voice input.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If cameras are used to track facial expressions, then facial expression tracking accuracy is improved, but device complexity and user comfort deteriorate due to the need for additional hardware in VR/AR devices
Solution Approach 1:
The patent extracts the facial expression tracking function from camera-based hardware and relocates it to software-based voice signal processing. By removing the camera dependency and using only voice input, the system achieves facial expression tracking without adding complex hardware to VR/AR devices.
Solution Approach 2:
The patent replaces the mechanical/optical camera-based tracking system with an acoustic signal processing system. Voice signals are processed through neural networks to generate facial expression coefficients, substituting physical camera hardware with software-based acoustic analysis.
2Measurement precision
If cameras are used to track facial expressions, then facial expression tracking capability is improved, but ease of operation deteriorates as users cannot wear VR/AR devices comfortably with external cameras
Solution Approach 1:
The patent removes the camera component from the VR/AR device configuration and replaces it with voice-based processing. Users can wear the device comfortably without external cameras, as the system uses only the microphone input already present in modern VR/AR headsets.
Solution Approach 2:
The system uses the voice input that users naturally produce during normal conversation, requiring no additional sensors or hardware. The VR/AR device leverages its existing microphone to capture voice signals for facial expression synthesis, making the system both comfortable and convenient for users.
3Measurement precision
If voice signals are processed through neural networks with multiple layers, then facial expression prediction accuracy is improved, but computing power requirements increase
Solution Approach 1:
The patent segments the voice signal processing into distinct stages: voice signal input, chunking into segments, neural network processing to generate intermediate values, and final facial expression coefficient generation. This segmentation allows for optimized processing at each stage, balancing accuracy with computational efficiency.
Solution Approach 2:
The patent performs preliminary processing of voice signals by chunking them into segments before neural network input. This preprocessing step organizes the data in a way that facilitates more efficient neural network processing, reducing the computational burden while maintaining prediction accuracy.
Data Source
AI summary
A Method for Implementing Voice-Based Avatar Facial Expression. The method comprises: obtaining, by a processor included in a computing device, chunks based on a voice signal, wherein the chunks comprise an amplitude information of the voice signal for a particular time, inputting input signals corresponding to the chunks into a neural network, computing, by a layer included in the neural network, a value reflecting a relationship between at least two of the input signals based on the input signals, predicting, by an output layer included in the neural network, facial expression coefficients by outputting output signals based on the value and implementing an expression of a predetermined avatar based on the facial expression coefficients. At least two of the output signals are generated by reflecting a relationship between at least two of the input signals.