Real-time 3D Body Pose Estimation via Two-Stage Neural Networks

Solution Overview

Problem

Conventional methods for body pose estimation in unconstrained video sequences are not computationally efficient for real-time applications, especially on hardware-limited devices, and often fail to provide complete 3D pose estimates due to occlusions and out-of-frame body parts.

Innovation Solution

A computer-implemented system and method that uses a two-stage neural network approach to track body features and estimate poses in real-time. The first neural network detects 2D keypoints, and the second network provides depth estimation values, which are then processed to generate 3D coordinates and fit a skeleton model for pose estimation.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If conventional body tracking methods are used, then 2D positional information can be obtained, but 3D pose estimation is not provided

Engineering Contradiction:

Improve3D pose estimation accuracyVSAvoidsystem complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The system segments the body tracking problem into distinct components: 2D keypoint detection from images, depth estimation from 2D coordinates, and 3D pose reconstruction from 2D and depth data. Each component is handled by specialized processing modules that work together to achieve complete 3D pose estimation without requiring a single complex system

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system transitions from 2D image coordinates to 3D pose estimation by introducing depth as an additional dimension. Depth values are estimated from 2D keypoint coordinates and then combined with the 2D data to reconstruct full 3D body pose, enabling applications that require three-dimensional spatial understanding

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Productivity

If conventional methods are applied to real-time video, then processing can be performed, but computational efficiency is insufficient for hardware-limited devices

Engineering Contradiction:

Improvereal-time processing speedVSAvoidcomputational energy consumption

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The system performs preliminary 2D keypoint detection and depth estimation before final 3D pose reconstruction. By pre-processing the data in this manner and using optimized algorithms for each stage, the system reduces the computational burden on hardware-limited devices while maintaining real-time processing capability

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system replaces traditional mechanical or physics-based pose estimation methods with neural network-based approaches. The trained neural networks perform 2D keypoint detection and depth estimation more efficiently than conventional algorithms, reducing computational energy consumption while improving accuracy on mobile devices

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Reliability

If body parts are out of frame or occluded, then complete 3D pose estimation cannot be provided, but user freedom is restricted

Engineering Contradiction:

Improvepose estimation completenessVSAvoidenvironmental adaptability

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The system performs preliminary 2D keypoint detection and depth estimation before final 3D pose reconstruction. By pre-processing the data in this manner and using optimized algorithms for each stage, the system reduces the computational burden on hardware-limited devices while maintaining real-time processing capability

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system replaces traditional mechanical or physics-based pose estimation methods with neural network-based approaches. The trained neural networks perform 2D keypoint detection and depth estimation more efficiently than conventional algorithms, reducing computational energy consumption while improving accuracy on mobile devices

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS12299930B2Real-time body pose estimation system and method in unconstrained video

Publication Date: 2025.05.13 IMAGE METRICS LTD

AI summary

In one or more implementations, body features of a subject can be tracked and a pose estimation produced using an unconstrained video sequence. The video sequence constitutes a physical record of the body features of the subject. A set of 2D coordinates corresponding to the body features of the subject can be received from a first neural network and 2D Keypoints are detected and transmitted to a second neural network. The second neural network can return corresponding depth estimation values, which can be processed to generate a set of 3D coordinates that correspond to the body features of the subject in camera space. Furthermore, one or more non-linear optimizations can be applied to fit a predetermined skeleton to the generate 3D Keypoints, and produce the pose estimation corresponding to the body features of the subject.