Model-Fused Depth Maps for Surgical Instrument Augmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer-assisted surgical systems face challenges in accurately determining depth information for surgical spaces, often resulting in unreliable or missing depth data, particularly at object edges and near the camera viewpoint, which hinders effective augmentation of surgical imagery.
Innovation Solution
A system that uses model data, such as CAD models, to align with the pose of surgical instruments in the surgical space, generating an enhanced depth map that includes both sensed and modeled depth information to fill in missing or unreliable data, ensuring accurate depth values for improved augmented imagery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If depth information is determined using captured imagery alone, then the system can operate with simple processing, but the depth information becomes unreliable or missing at object edges and near the camera viewpoint
Solution Approach 1:
The patent combines captured imagery data with pre-existing 3D model data to generate enhanced depth maps. The system merges depth information from both sources, using the model data to fill in missing or unreliable depth values at object edges and near the camera viewpoint, while using captured imagery for general depth estimation. This combination resolves the contradiction by improving reliability through multi-source data fusion without requiring a completely complex processing system.
Solution Approach 2:
The patent introduces a depth map generation system that acts as an intermediary between captured imagery and the final augmented reality display. This intermediary process takes the raw depth data from captured imagery, enhances it using 3D model information, and produces improved depth maps that are then used for rendering. The intermediary system bridges the gap between simple imaging and complex rendering requirements.
2Loss of information
If model data is used to align with object pose and generate enhanced depth maps, then depth information completeness improves, but the system complexity increases
Solution Approach 1:
The patent uses pre-existing 3D models of surgical instruments that are prepared in advance. These models contain known geometric and depth information that can be directly applied when the instrument appears in the captured imagery. By performing the complex 3D modeling work beforehand, the system avoids the need to compute complete depth information from scratch during surgery, reducing real-time processing complexity while improving depth completeness.
Solution Approach 2:
The system creates a virtual copy of the surgical instrument from the captured imagery by matching visual features with the pre-existing 3D model. Once matched, the system copies the depth information from the model to the corresponding regions in the captured image, filling in missing depth data without requiring complex real-time depth computation for the entire instrument structure.
Data Source
AI summary
An exemplary processing system of a computer-assisted surgical system identifies an object in a surgical space, accesses a model of the object, identifies a pose of the object in the surgical space, aligns the model with the pose of the object to define depth data for the model in the surgical space, and generates a depth map in the surgical space for at least a portion of the object based on the depth data for the model. Corresponding systems and methods are also described.