Real-time 3D Pose Estimation via Segmented Machine Learning Pipeline
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current human pose estimation technologies lack the capability to generate and display a digital twin of a user in real-time within augmented, virtual, and mixed reality environments, which is essential for applications like fitness and gesture control.
Innovation Solution
A system and method that processes 2D image data using machine learning algorithms to estimate 3D key point coordinates, transform them into a rotation-based representation, and transmit this information in real-time for rendering a digital twin on a displaying module, such as a smartphone or AR glasses, using a data processing and displaying module.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If human pose estimation is performed using traditional machine learning algorithms, then the processing speed is limited, but real-time processing is required for digital twin generation
Solution Approach 1:
The system segments the pose estimation process into distinct stages: 2D key point detection from image data, 3D key point estimation from 2D coordinates, and digital twin generation from 3D coordinates. This segmentation allows each stage to be optimized independently, with machine learning algorithms applied selectively to critical path operations, thereby improving overall processing speed while maintaining accuracy.
Solution Approach 2:
The system performs preliminary processing by first extracting 2D key point coordinates from image data before estimating 3D coordinates. This preliminary 2D detection step prepares data in advance for the more computationally intensive 3D estimation, enabling real-time processing by reducing the complexity of the main estimation task.
2Measurement precision
If complex machine learning algorithms are used for accurate pose estimation, then measurement precision improves, but computational resources increase
Solution Approach 1:
The pose estimation pipeline is segmented into multiple stages with progressively increasing computational complexity. Simple 2D key point detection is performed first, followed by 3D estimation only for detected key points. This segmentation reduces overall computational resource consumption compared to applying complex algorithms to the entire image, while maintaining accuracy through focused application of advanced techniques.
Solution Approach 2:
The system applies machine learning algorithms selectively to extract only the necessary 2D key point coordinates from image data, rather than processing the entire image. This partial action approach reduces computational resource consumption by focusing processing power only on relevant regions and features needed for accurate pose estimation.
3Productivity
If 3D key point coordinates are estimated directly from 2D image data, then real-time processing is achieved, but manufacturing precision of the digital twin decreases
Solution Approach 1:
The coordinate transformation process is segmented into distinct steps: 2D key point extraction from image data, 3D key point estimation from 2D coordinates, and digital twin construction from 3D coordinates. This segmentation enables real-time processing by breaking down the complex transformation into manageable stages, while maintaining manufacturing precision through careful optimization of each individual stage.
Solution Approach 2:
The system performs preliminary 2D key point detection and 3D estimation before final digital twin generation. This preliminary processing prepares accurate coordinate data in advance, ensuring that when the digital twin is constructed in real-time, the manufacturing precision is maintained through the use of pre-computed accurate 3D coordinates.
Data Source
Figure 1
Figure 2
Figure 3~4a
AI summary
A system for monitoring a body pose of a user is disclosed, the system comprising a data processing module, which is configured to perform in real-time or quasi real-time the steps of: receiving 2D image data of the user performing the body pose, detecting the body of the user on the 2D image data, extracting 2D key point coordinates of the detected body from the 2D image data using machine learning algorithms, estimating 3D key point coordinates from the extracted 2D key point coordinates using machine learning algorithms, transforming the estimated 3D key point coordinates into a rotation-based representation of the user, and outputting the rotation-based representation of the user.