Avatar Control via Mobile Sensor Data Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial reality (XR) systems lack the ability to accurately infer user actions, such as facial expressions and emotional states, from non-monitored aspects, limiting the realism of avatar representations and requiring specialized hardware for user input.
Innovation Solution
An avatar control system that utilizes non-XR devices like mobile phones, incorporating sensors, microphones, and cameras to infer user behavior, generate motion plans, and apply kinematic rules to create realistic and customizable avatar animations, allowing users to control avatars with external devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional XR systems use specialized XR input devices to track user movements, then the tracking precision is improved, but the device complexity and requirement for specialized hardware increases
Solution Approach 1:
The patent uses a mobile device as an intermediary to capture user movements and transmit them to the XR system. The mobile device acts as a mediator between the user and the XR system, allowing control without specialized XR input devices. The mobile device's sensors (accelerometer, gyroscope, magnetometer) capture movement data which is then processed and transmitted to control the avatar in the XR environment.
Solution Approach 2:
The patent enables universal control of XR avatars using a mobile device that serves multiple functions: movement tracking, gesture recognition, and communication with the XR system. This multi-functional approach eliminates the need for dedicated XR controllers, allowing a single device to perform multiple control functions.
2Ease of operation
If XR systems use basic control devices for avatar movement, then the ease of operation is improved, but the ability to infer user actions and emotional states deteriorates
Solution Approach 1:
The patent analyzes multiple parameters from mobile device sensors (acceleration, rotation, magnetic field) to infer user actions and emotional states. By examining changes in these parameters over time and their combinations, the system can determine not just movement but also the user's emotional state, maintaining information richness while using a simple mobile device interface.
Solution Approach 2:
The system provides feedback by mapping mobile device movements to avatar actions in real-time. The avatar reflects the user's movements and inferred emotional states, creating a feedback loop that enhances the expressive capability of the simple mobile device control interface.
3Measurement precision
If XR systems require specialized hardware for accurate avatar control, then the measurement precision is improved, but the adaptability and accessibility deteriorates
Solution Approach 1:
The patent makes the XR system accessible to anyone with a mobile device, eliminating the need for specialized XR hardware. The mobile device's existing sensors are utilized for precise movement detection, making the system universally accessible while maintaining measurement precision through sophisticated sensor data processing.
Data Source
AI summary
Controlling an avatar in an XR environment can be based on input(s) from external, non-XR computing devices, such as mobile devices. An external device can capture sensor data, audio data, and/or image data, which are processed by an avatar control system to determine a gesture performed by the user, gauge the level of intensity or excitement of the user, and/or infer the user's sentiment or emotional state. The avatar control system can generate a motion plan based on these inferences drawn from the external device sensor data based on a baseline animation or motion template. The system can generate avatar animations that have exaggerated or understated expressions relative to the user's real-world gestures, possibly exceeding limitations of a kinematic model that otherwise constrain the avatar's animations. The technology can enable users to control an avatar in an XR environment without specialized XR equipment.