3D Model Fusion for Real-Time Laparoscopic Depth Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing laparoscopic surgical procedures face challenges in aligning preoperative 3D virtual models with intraoperative 2D video images due to complex manual operations and high sensitivity, particularly with fast endoscope movements, leading to a lengthy and disruptive alignment process.
Innovation Solution
A method and system for aligning a 3D virtual model with a 2D laparoscopic image using key-pose generation, feature vocabulary construction, and continuous motion-tracking, enabling automatic or semi-automatic determination of 3D pose adjustments to achieve a matching overlay.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual operation is used to establish initial alignment between 3D model and 2D video images, then alignment accuracy can be achieved, but the procedure becomes lengthy and disruptive to surgical intervention
Solution Approach 1:
The system performs preliminary actions by automatically detecting landmarks and pre-computing transformation parameters before the surgical procedure begins. The key-pose generation and feature vocabulary construction are completed in advance, so that when alignment is needed during surgery, the system can quickly apply pre-prepared transformations rather than performing manual landmark matching from scratch.
Solution Approach 2:
The system enables self-service by automatically performing the alignment process without requiring manual intervention. The automated landmark detection, feature matching, and transformation computation are executed by the system itself using computer vision algorithms, eliminating the need for physicians to manually operate controls during the alignment process.
2Manufacturing precision
If mouse-based control is used to modify 6-DOF transformation, then precise adjustment can be achieved, but the operation becomes complex and sensitive
Solution Approach 1:
The system replaces the mechanical mouse-based control system with an automated computational system. Instead of requiring physicians to manually manipulate 6-DOF transformation parameters through mouse operations, the system uses computer vision algorithms to automatically compute the transformation matrix that aligns the 3D model with the 2D video images, substituting mechanical interaction with algorithmic processing.
Solution Approach 2:
The system introduces an intermediary computational layer between the physician's intent and the transformation application. The automated landmark detection and feature matching systems act as intermediaries that translate the visual information from video frames into precise transformation parameters, eliminating the need for direct manual manipulation of complex 6-DOF controls.
3Speed
If fast endoscope movements are used to capture surgical scenes, then real-time imaging is achieved, but alignment becomes more difficult and blurred
Solution Approach 1:
The system performs preliminary action by pre-construction of feature vocabularies from training images before fast endoscope movements occur. These pre-computed feature descriptors and landmark positions are stored and ready for rapid matching during surgery, allowing the system to quickly align images even when captured at high speeds with motion blur.
Solution Approach 2:
The system applies parameter changes by adjusting the robustness parameters of the feature matching algorithm to accommodate fast movements. The automated detection system uses modified parameter thresholds and blur-tolerant feature descriptors that maintain alignment precision even when images are captured during rapid endoscope motion, allowing the system to adapt to varying image quality conditions.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present teaching relates to method, system, medium, and implementations for estimating 3D coordinate of a 3D virtual model. Two pairs of feature points are obtained. Each of the pairs includes a respective 2D feature point on an organ observed in a 2D image, acquired during a medical procedure, and a respective corresponding 3D feature point from a 3D virtual model, constructed for the organ prior to the procedure based on a plurality of images of the organ. The first and the second 3D feature points have different depths. A 3D coordinate of a 3D feature point is determined based on the pairs of feature points so that a projection of the 3D virtual model from the 3D coordinate substantially matches the organ observed in the 2D image.