Motion Retargeting via Neural Network Cycle Consistency

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing computer-modeling systems for retargeting motion between different skeletons are inaccurate, inefficient, and inflexible, requiring human intervention and relying on limited and unreliable data sets, which leads to tedious and costly processes.

Innovation Solution

A neural network system using a motion synthesis neural network with an encoder recurrent neural network, a decoder recurrent neural network, and a forward kinematics layer, trained with adversarial loss and cycle consistency loss to generate accurate and realistic motion sequences for target skeletons.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Device complexity

If conventional computer-modeling systems directly map coordinates for joints of source skeleton to joints of standard skeleton, then the retargeting process is simplified, but accuracy deteriorates due to assumptions about end-effector positions and segment lengths

Engineering Contradiction:

Improveretargeting process complexityVSAvoidmotion retargeting accuracy

Core Design Contradiction:

Device complexityVSManufacturing precision

Solution Approach 1:

The patent replaces conventional mechanical coordinate mapping systems with a neural network-based system. The neural network learns the complex non-linear relationships between source and target skeleton joints through training, eliminating the need for rigid coordinate transformations and hand-crafted mapping algorithms, thereby achieving higher accuracy without proportional increase in system complexity

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent transforms the retargeting approach by changing from fixed coordinate mapping parameters to learned parameters through neural network training. The system adapts parameters dynamically based on the specific skeleton pairs and motion characteristics, allowing accurate retargeting across diverse skeletons with different segment lengths and joint positions

Inventive Principle:

Principle #35Parameter changes

2Reliability

If machine-learning models use hand-designed objectives for end-effectors, then motion essence can be preserved, but reliability deteriorates due to dependence on human-discovered properties

Engineering Contradiction:

Improvemotion property transfer reliabilityVSAvoidhuman intervention requirement

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent implements self-service by enabling the neural network to automatically learn motion properties directly from paired motion data without human intervention. The system discovers important features and properties of motion and skeletons autonomously during training, eliminating dependence on human-discovered properties while improving reliability through data-driven learning

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent incorporates feedback mechanisms where the neural network learns from the discrepancy between predicted and actual motion outcomes. Through adversarial training and cycle consistency loss, the system receives continuous feedback to refine its understanding of motion properties, progressively improving reliability without human input

Inventive Principle:

Principle #23Feedback

3Measurement precision

If paired motion data for different skeletons is used for training, then supervised machine-learning approaches can be implemented, but feasibility deteriorates due to limited and difficult-to-generate data sets

Engineering Contradiction:

Improvesupervised learning accuracyVSAvoiddata set applicability

Core Design Contradiction:

Measurement precisionVSAdaptability or versatility

Solution Approach 1:

The patent enhances universality by training the neural network on diverse paired motion data from multiple skeleton pairs. The learned model becomes adaptable to different skeleton types and motion characteristics, enabling broad applicability across various retargeting scenarios while maintaining precision through the universal patterns learned from diverse training data

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS10546408B2Retargeting skeleton motion sequences through cycle consistency adversarial training of a motion synthesis neural network with a forward kinematics layer

Publication Date: 2020.01.28 ADOBE INC

AI summary

This disclosure relates to methods, non-transitory computer readable media, and systems that use a motion synthesis neural network with a forward kinematics layer to generate a motion sequence for a target skeleton based on an initial motion sequence for an initial skeleton. In certain embodiments, the methods, non-transitory computer readable media, and systems use a motion synthesis neural network comprising an encoder recurrent neural network, a decoder recurrent neural network, and a forward kinematics layer to retarget motion sequences. To train the motion synthesis neural network to retarget such motion sequences, in some implementations, the disclosed methods, non-transitory computer readable media, and systems modify parameters of the motion synthesis neural network based on one or both of an adversarial loss and a cycle consistency loss.