Optical Tracking and 3D Surface Acquisition for Surgical AR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional optical tracking systems in surgical environments face challenges with lower accuracy and reliability when using visible light due to CPU-intensive image analysis and less reliable fiducial marker detection, leading to lower update rates and inaccuracies, especially in computer-assisted surgery systems.
Innovation Solution
Employing RGB color sensors and structured light modules that project patterns in the NIR spectrum, capturing images with both RGB and NIR sensors to enhance tracking accuracy and reconstruct 3D surfaces, allowing for improved registration and augmented reality overlays.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If visible light image sensors are used for optical tracking, then color information and image-based tracking are enabled, but tracking accuracy and reliability decrease due to CPU-intensive processing and less reliable fiducial marker detection
Solution Approach 1:
The patent divides the tracking system into two separate sensor channels: visible light sensors for capturing color images and providing adaptability, and NIR sensors for performing high-precision optical tracking. This segmentation allows each sensor type to specialize in its optimal function, with the NIR channel handling fiducial marker detection and position tracking while the visible channel provides color information for surgical visualization.
Solution Approach 2:
The patent introduces NIR light sources and NIR-pass filters as intermediaries to enable high-precision tracking. The NIR illumination sources emit infrared light that reflects off fiducial markers, and the NIR sensors with corresponding filters detect these reflections to determine marker positions with high accuracy, independent of the visible light imaging channel.
2Adaptability or versatility
If full image analysis is performed to detect fiducial markers, then color images can be utilized for tracking, but update rate decreases due to CPU-intensive processing
Solution Approach 1:
The patent extracts the fiducial marker detection function from the full visible light image analysis process by implementing a separate NIR detection channel. The NIR sensors are specifically configured to detect only the reflected NIR light from fiducial markers, allowing marker position detection to occur independently and simultaneously with color image capture, thereby maintaining high update rates without requiring CPU-intensive full image analysis.
3Measurement precision
If NIR-enhanced CMOS image sensors are used, then quantum efficiency in NIR spectrum is improved, but the system becomes more complex and costly
Solution Approach 1:
The patent employs standard monochrome CMOS image sensors that can detect both visible light and NIR wavelengths, eliminating the need for specialized NIR-enhanced sensors. By utilizing the inherent dual-spectrum sensitivity of conventional sensors, the system achieves NIR detection capability without increasing device complexity or cost, while still maintaining the ability to capture color images through separate RGB sensors.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Enhances tracking accuracy and reliability, enabling more efficient and precise surgical procedures by providing real-time tracking and augmented reality overlays, thus improving surgical workflow and reducing registration time.
Implementation Method 1
projecting a pattern with a structured light module, the pattern being projected in a near-infrared (NIR) spectrum
Implementation Method 2
A second image is captured with the image sensor... to determine tracking information including at least a three-dimensional (3D) position of each of one or more fiducials
Implementation Method 3
Once these points are detected, they are processed in a manner similar to NIR-based systems (e.g., triangulation, pose estimation, etc.)
Data Source
Figure 1
Figure 2
Figure 3A~3C
AI summary
A computer assisted system is disclosed that includes an optical tracking system and one or more computing devices. The optical tracking system includes an RGB sensor and is configured to capture color images of an environment in the visible light spectrum and tracking images of fiducials in the environment in a near-infrared spectrum. The computer assisted system is configured to generate a color image of the environment using the color images, identify fiducial locations using the tracking images, generate depth maps from the color images, reconstruct three-dimensional surfaces of structures based on the depth maps, and output a display comprising the reconstructed three-dimensional surface and one or more surgical objects that are associated with the tracked fiducials. The computer assisted system can further include a monitor or a head-mounted display (HMD) configured to present augmented reality (AR) images during a procedure.