TOF Camera Pose Estimation via 3D Point Cloud and Skeleton Model
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for estimating human pose in real-time from image sequences using 2D or 3D data face challenges such as complex background segmentation, high processing time, and inability to detect movements like folded arms, especially with rigid object-based approaches.
Innovation Solution
A method utilizing a time-of-flight (TOF) camera to generate 3D aggregate points and fit a flexible skeleton model with a self-organizing map (SOM) for real-time pose estimation, allowing continuous updating of node positions and maintaining neighborhood structure for accurate detection of arm movements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If 2D image data is used for pose estimation, then processing speed is improved, but segmentation difficulty increases and processing power is sacrificed
Solution Approach 1:
The patent introduces a 3D aggregate of points as an intermediary representation that bridges 2D image data and 3D pose estimation. This intermediate structure allows the system to leverage 2D processing speed while achieving 3D pose accuracy through the aggregated point cloud representation, resolving the contradiction between processing speed and segmentation difficulty.
Solution Approach 2:
The patent transforms 2D image data into a 3D aggregate of points, adding the depth dimension enabled by TOF camera technology. This dimensional transformation allows the system to maintain 2D processing efficiency while gaining 3D spatial information, thereby improving segmentation accuracy without sacrificing processing speed.
2Measurement precision
If 3D volume acquisition methods are used, then pose estimation accuracy is improved, but processing time increases
Solution Approach 1:
The patent extracts only the essential 3D surface information needed for pose estimation from the full 3D volume data. By using TOF camera depth maps to generate a 3D aggregate of points representing only the visible surface, the system avoids processing the entire 3D volume, thereby reducing processing time while maintaining pose estimation accuracy.
Solution Approach 2:
The patent applies partial action by processing only the necessary portion of 3D data (surface points) rather than the complete 3D volume. This selective processing approach maintains sufficient accuracy for pose estimation while significantly reducing computational requirements and processing time.
3Device complexity
If rigid object models are used for pose estimation, then computation is simplified, but detection of complex movements like folded arms becomes impossible
Solution Approach 1:
The patent transitions from rigid object models to a dynamic skeleton model where nodes and edges can adapt to complex movements. The skeleton model with adjustable nodes and edges can accommodate folded arms and other complex poses, while the TOF-based 3D aggregate provides the flexibility needed to represent non-rigid body configurations.
Solution Approach 2:
The patent changes the model parameters from fixed rigid structures to adjustable skeleton parameters. The skeleton model with variable node positions and edge configurations allows the system to adapt to complex movements like folded arms, maintaining computational efficiency while improving versatility.
4Measurement precision
If stereoscopy is used for 3D pose estimation, then depth information is improved, but processing time increases and texture requirements are stringent
Solution Approach 1:
The patent replaces the mechanical stereoscopy system with a TOF (time-of-flight) camera system. This substitution allows direct acquisition of depth information through optical means, eliminating the complex image processing required by stereoscopy and significantly reducing processing time while maintaining accurate depth data.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Enables reliable pose estimation at up to 25 frames per second on a standard PC with minimal computing resources, effectively handling complex movements and backgrounds, and is scalable for various movable objects.
Implementation Method 1
The frames of the image sequence have been recorded by a time-of-flight (TOF) camera so that they can be processed by a computer and have brightness and distance data as functions of pixel coordinates of the camera for each frame of the sequence
Data Source
AI summary
The invention relates to a method for the real-time-capable, computer-assisted analysis of an image sequence of an object consisting of elements that can be moved relative to each other and are interconnected, said sequence containing a variable pose, wherein the individual images of the image sequence are recorded by way of a time-of-flight (TOF) camera such that said images can be processed by a computer, and contain brightness and distance data as functions of the pixel coordinates of the camera for each image of the sequence, comprising the following steps: a. Capturing the pixels of an individual image forming the object, b. calculating a three-dimensional (3D) point cloud in a virtual space, said point cloud representing the surface of the object that is visible to the camera, by a computational projection of object-depicting pixels in such a space, while taking captured distance data to the object into consideration, c. fitting a model of the object consisting of nodes and edges into the computer-generated 3D point cloud for the individual images, wherein the nodes represent a selection of elements of the object and the edges represent the connections of said elements amount each other, d. iteratively updating all node positions by applying a learning rule for training a self-organizing map having a previously defined number of randomly selected dots of the point cloud, e. repeating steps a. to d. for each subsequent individual image of the sequence, wherein for the fitting in step c. the result of step e. of the preceding image is used in each case, and f. determining the varying pose from the positions of predetermined nodes of the model which have been captured in at least representative images of the image sequence.