3D Body-Part Registration Using Monocular Depth on One Display
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems for registering a 3D model of a body part with a real-world image in minimally invasive surgery require extensive hardware, are space-occupying, costly, and suffer from registration errors due to assumptions about the stability of the surgical instrument's position and the target organ's shape, necessitating complex calibration and re-registration during procedures.
Innovation Solution
A device and method utilizing a user input device like a wireless 3D mouse to register a 3D model with a real-world image on a single display, incorporating monocular depth estimation and topological pattern analysis to determine a registration matrix, eliminating the need for tracking systems and reducing hardware requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional navigation systems with optical tracking cameras and multiple monitors are used, then spatial anatomical information can be provided to the surgeon, but the system becomes space-occupying, costly, and complex
Solution Approach 1:
The patent combines the 3D model display and real-world image display onto a single monitor, merging previously separate visualization functions. This eliminates the need for multiple monitors while maintaining the ability to provide spatial anatomical information, directly reducing system complexity and installation space
Solution Approach 2:
The patent extracts and eliminates the optical tracking camera and tracking markers from the system. Instead of using complex hardware tracking, the system uses a user input device to manually indicate correspondences between the 3D model and real-world images, removing unnecessary complex components while preserving registration functionality
2Measurement precision
If tracking markers are attached to surgical instruments and optical tracking is used, then instrument position can be tracked, but the system requires continuous field of view and frequent re-registration
Solution Approach 1:
The system uses the surgical instrument itself (or structures already present in the surgical field) as reference points for registration, rather than requiring external tracking markers. The reference points are naturally part of the surgical anatomy or instruments, eliminating the need for separate tracking hardware and reducing field of view constraints
Solution Approach 2:
The registration is performed preliminarily by indicating correspondences between the 3D model and real-world images before the surgical procedure begins. Once registered, the system maintains alignment without requiring continuous tracking or re-registration during surgery, eliminating field of view constraints
3Loss of information
If multiple monitors are used to display 3D models and real-world images separately, then spatial information can be visualized, but installation space and system complexity increase
Solution Approach 1:
The patent merges the display of 3D models and real-world images onto a single monitor by superimposing them in a registered state. This integration maintains complete spatial information visualization while eliminating the need for multiple separate displays, directly reducing installation space and system complexity
Solution Approach 2:
The patent uses depth information from 3D data to enhance the 2D display, allowing spatial relationships to be visualized on a flat screen. By incorporating depth cues and stereoscopic display capabilities, the system maintains three-dimensional spatial information on a two-dimensional display surface
Data Source
AI summary
A device including a processor that receives 3D data of a model and 2D data of a real-world image of a body part, visualize the model and the body part on a 2D display, receive user input data, and operate a mouse pointer on the 2D display based on the user input data, receive user input data, which, for at least a first to third pair comprising a reference point and a corresponding reference point indicates a registration between the reference point in the real-world image and the corresponding reference point in the 3D model, expand the 2D data of the reference points into synthesized 3D data by adding depth information from one or more of the 3D data and the 2D data, determine a registration matrix for registering the 3D data and the synthesized 3D data using the 3D coordinates of the first to third pair of reference points.