Partial Body 3D Modeling with Attention-Graph Networks for Occlusion Handling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing medical imaging systems face challenges in accurately determining patient positioning and motion during scans due to partial occlusions by individuals or objects, leading to difficulties in generating precise 3D body models.
Innovation Solution
An apparatus using an artificial neural network with a self-attention module and graph convolution module to detect body parts in images, determining classification labels and vertices, and generating 3D models based on these features, allowing for accurate estimation of partial and full-body 3D models even with partial occlusions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If template-based methods are used to address partial occlusion, then a consistent model topology is produced, but the shape and fine details of the body part are not accurately captured
Solution Approach 1:
The patent segments the body model into multiple body parts, each with its own dedicated neural network processor. This allows each body part to be modeled independently with high precision while maintaining overall model consistency through the hierarchical structure.
Solution Approach 2:
The patent transitions from 2D image data to 3D body models by processing images from multiple views and reconstructing three-dimensional representations. This dimensional transformation enables accurate capture of body part shapes and details that cannot be obtained from single 2D views.
2Manufacturing precision
If multi-view stereo reconstruction methods are used to produce a complete model, then a dense model is generated, but the system becomes sensitive to occlusions and requires many views
Solution Approach 1:
The patent introduces neural networks as intermediary processing layers between the sensor inputs and the final 3D model reconstruction. These neural networks learn to infer occluded body parts from visible portions, reducing the number of required sensor views and making the system more robust to occlusions.
Solution Approach 2:
The patent performs preliminary processing of images from multiple views through neural networks before final model fusion. This preliminary action includes detecting body parts, estimating their 3D positions, and predicting occluded regions, which simplifies the subsequent model fusion process and reduces the number of required views.
3Measurement precision
If external perception sensors are used to determine patient pose and keypoints, then positioning information is obtained, but accurate estimation becomes difficult when occlusions are present
Solution Approach 1:
The patent implements feedback mechanisms where the neural networks continuously refine their estimates of occluded body parts based on the detected visible portions and the learned relationships between different body parts. This feedback loop enables accurate reconstruction of patient pose and keypoints even when parts of the body are occluded.
Solution Approach 2:
The patent creates virtual copies or representations of occluded body parts by leveraging the detected visible portions and the learned 3D body model structure. These copied representations allow the system to estimate patient positioning and motion accurately without direct observation of all body parts.
Data Source
AI summary
An apparatus may obtain an image that depicts a first human body part and determine, based on the image, a classification label and a plurality of vertices associated with the first human body part. The classification label may indicate a class of the first human body part and the plurality of vertices may correspond to points of the first human body part in a three-dimensional (3D) space. The determinations may be made using an artificial neural network that includes a self-attention module and a graph convolution module. The apparatus may generate a first 3D model representative of the first human body part based at least on the plurality of vertices associated with the first human body part. The self-attention module may determine global features of the image indicating an interrelationship of the plurality of vertices and the graph convolution module may refine the global features determined by the self-attention module.