Body Pose Estimation Using Masked Pixel-to-Model Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing instance segmentation methods, such as Mask R-CNN, suffer from low resolution masks and high inference times, making them impractical for real-time applications like autonomous driving, especially in scenarios with multiple instances and occlusions.
Innovation Solution
A direct and end-to-end pose estimation method using a divide-and-conquer approach that predicts masks and representations simultaneously, allowing parallel processing and reducing computational dependence on the number of instances, with features extracted as a pyramid and transformed into a sparse format for efficient computation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a top-down detect-then-segment approach is used (e.g., Mask R-CNN), then object detection accuracy is improved, but inference time increases and frame rate decreases
Solution Approach 1:
The patent inverts the traditional top-down detect-then-segment approach by implementing a bottom-up segment-then-detect methodology. Instead of first detecting objects with bounding boxes and then generating masks, the system first generates segmentation masks for all regions and then performs detection on the segmented instances. This inversion eliminates the sequential dependency that causes inference time delays, allowing parallel processing of segmentation and detection operations, thereby improving frame rate while maintaining detection accuracy.
Solution Approach 2:
The patent applies segmentation by dividing the image processing task into independent per-pixel classification operations. Each pixel is processed independently to determine its instance assignment, allowing parallel computation across the entire image. This segmentation approach eliminates the bottleneck of sequential bounding box detection and mask generation, enabling real-time processing speeds while maintaining precise object delineation.
2Measurement precision
If bounding box detection is performed first, then object localization is improved, but early commitment errors cannot be recovered and overlap ambiguity increases
Solution Approach 1:
The patent inverts the detection pipeline by performing segmentation before final object localization. Instead of committing to bounding boxes early in the process, the system first segments the image into instance masks and then localizes objects based on these masks. This inversion allows the system to avoid early commitment errors, as the segmentation step provides more accurate instance boundaries that can be used to correct localization errors in subsequent steps, thereby improving detection robustness.
Solution Approach 2:
The patent extracts the segmentation task from the traditional detect-then-segment pipeline and performs it independently as the first step. By extracting segmentation as a separate, initial operation that processes all pixels in parallel, the system eliminates the constraint of bounding box-based localization. This allows the system to directly compute instance masks without being constrained by preliminary bounding box predictions, reducing overlap ambiguity and improving the ability to recover from localization errors.
3Measurement precision
If high resolution masks are generated, then pose estimation accuracy is improved, but computational complexity and memory usage increase
Solution Approach 1:
The patent segments the image processing task into independent per-pixel operations that can be executed in parallel. By formulating instance segmentation as a per-pixel classification problem where each pixel is independently classified according to its instance assignment, the system achieves high-resolution mask generation without sequential processing bottlenecks. This segmentation approach allows the computational workload to be distributed across multiple processors or GPU cores, maintaining high pose estimation accuracy while reducing overall computational complexity through parallelization.
4Measurement precision
If instance segmentation is performed for every detected object, then pose estimation accuracy is improved, but inference time scales unfavorably with the number of instances
Solution Approach 1:
The patent segments the processing task at the pixel level, allowing all pixels in the image to be processed simultaneously in a single pass. Instead of performing separate segmentation operations for each detected object instance, the system performs a unified per-pixel classification that assigns every pixel to its corresponding instance in parallel. This approach maintains high pose estimation accuracy for all instances while ensuring that inference time remains constant regardless of the number of instances detected, as the parallel processing capability scales with the number of pixels rather than the number of objects.
Data Source
Figure 1
Figure 2
AI summary
A computer-implemented method (10) for estimating a pose of a body in at least one image (12), comprising a mask prediction (20) of predicting a mask for every detected instance of the body in the at least one image (12), and a representation prediction (30) of predicting a representation comprising a map between pixels of body regions in the whole at least one image and points of a body model, wherein the method further comprises, for every detected instance, applying the mask to the representation to derive the pose of the body.