3D Keypoint Detection Using 2D Camera Triangulation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current 3D optical depth sensing cameras struggle to accurately determine the 3D position of obstructed anatomy during medical examinations, as they cannot 'see' through objects and provide only valid depth readings for the topmost object, limiting their effectiveness in obstructed scenarios.
Innovation Solution
A system comprising a camera system and a processor that obtains 2D images and triangulates object keypoint projections to determine 3D coordinates, using scanner variables to account for the movement of medical scanner parts and cameras, and employs a neural network to estimate keypoint positions even when obstructed, leveraging U-Net architecture for precise image processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If 3D optical depth sensing cameras are used to simplify and automate the positioning task, then automation and ease of operation are improved, but measurement precision deteriorates in obstructed scenarios because depth readings are only available for the topmost object
Solution Approach 1:
The patent introduces visible light cameras as an intermediary system to capture 2D images of the subject and setup. These 2D images serve as intermediate data that can be processed to estimate 3D positions of keypoints even when obstructed by coils or other objects, thereby maintaining measurement precision while preserving automation benefits
Solution Approach 2:
The system creates a 3D model or representation of the subject and setup based on 2D images from visible light cameras. This copied 3D representation allows determination of keypoint positions without requiring direct depth measurements from the cameras, overcoming the limitation of obstructed scenarios
2Measurement precision
If multiple cameras are used to improve 3D coordinate determination, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent transitions from relying solely on depth information (one dimension) to using 2D image data from visible light cameras. By processing 2D images through keypoint detection and triangulation algorithms, the system achieves 3D coordinate determination without requiring complex multi-camera depth sensing setups
Data Source
AI summary
A system for obtaining object keypoints for an object in a medical scanner, wherein the object keypoints are three dimensional, 3D, coordinates with respect to the medical scanner of pre-determined object parts. The system comprises a camera system for obtaining two dimensional, 2D, images of the object in the medical scanner, wherein the camera system comprises one or more cameras, and a processor. The processor is configured to obtain scanner variables from the medical scanner, wherein the scanner variables include the position of a part of the medical scanner which determines a relative position between the cameras in the camera system and the object. The processor determines object keypoint projections in 2D coordinates based on the 2D images from the camera system and determines the object keypoints in 3D coordinates by triangulating the object keypoint projections with respect to the camera system based on the scanner variables.