Occluded Key Point Detection Using Multi-Modal CNN Weight Adjustment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning-based object detection techniques struggle to accurately determine key anatomical structures like joint locations when they are partially or completely blocked in medical images, due to reliance on unoccluded body portions.
Innovation Solution
A convolutional neural network (CNN) that processes RGB images in conjunction with depth or thermal images, adjusting weights based on additional imaging properties to compensate for occlusions, using modality-aware convolution operations and weightage maps to enhance detection accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If deep learning-based object detection techniques are used to determine joint locations, then automated detection capability is improved, but detection accuracy deteriorates when key points are occluded
Solution Approach 1:
The patent introduces depth information as an additional dimension to the traditional 2D RGB image data. By capturing depth maps alongside RGB images and processing them through separate convolutional layers, the system creates a multi-dimensional feature representation that enables accurate joint detection even when key points are occluded in the visual spectrum, thus resolving the contradiction between automation and accuracy under occlusion.
Solution Approach 2:
The patent combines multiple imaging modalities (RGB images and depth maps) into a composite input representation for the neural network. This composite approach integrates complementary information from different sensing modalities, where depth data provides structural information that compensates for occlusions in RGB images, thereby maintaining high detection accuracy while preserving automated operation.
2Measurement precision
If multiple imaging modalities are integrated, then detection accuracy under occlusion is improved, but device complexity increases
Solution Approach 1:
The patent merges the processing of RGB images and depth maps into a unified convolutional neural network architecture. By combining the feature extraction capabilities for both modalities within a single network and integrating their outputs through element-wise multiplication, the system achieves accurate occluded key point detection while avoiding the complexity of multiple separate processing pipelines.
Solution Approach 2:
The patent designs a universal neural network architecture that processes both RGB and depth modalities through shared convolutional layers and pooling operations. This multi-functional design allows the same network structure to handle different imaging modalities, reducing overall system complexity while maintaining the ability to detect key points accurately under various occlusion conditions.
Data Source
AI summary
Image-based key points detection using a convolutional neural network (CNN) may be impacted if the key points are occluded in the image. Images obtained from additional imaging modalities such as depth and/or thermal images may be used in conjunction with RGB images to reduce or minimize the impact of the occlusion. The additional images may be used to determine adjustment values that are then applied to the weights of the CNN so that the convolution operations may be performed in a modality aware manner to increase the robustness, accuracy, and efficiency of key point detection.