Multi-Sensor Camera Fusion of Color and NIR Images for Surgery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing minimally invasive surgical systems face challenges in simultaneously displaying near-infrared (NIR) fluorescence images and visible-range images without degrading spatial resolution or requiring time-consuming mode switching, which hinders real-time visualization of surgical sites.
Innovation Solution
A method and system that utilizes a multi-sensor camera with correlated visible-range sensors to predict NIR-induced fluorescence information, allowing concurrent display of approximate full-color images with overlaid NIR fluorescence, using one sensor for both visible and NIR imaging, thereby maintaining spatial resolution and enabling real-time visualization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If separate cameras are used for visible and NIR imaging, then imaging functionality is improved, but device complexity and spatial resolution are degraded
Solution Approach 1:
The patent applies multi-functionality by enabling a single visible-range camera to perform both visible imaging and NIR fluorescence detection. The camera alternates between visible mode and NIR mode, with the NIR light source illuminating the surgical site during NIR mode. This allows one device to serve multiple imaging purposes, reducing overall system complexity while maintaining both visible and NIR imaging capabilities
Solution Approach 2:
The patent combines visible-range imaging and NIR fluorescence imaging into a single integrated system. The visible camera and NIR light source are merged into one imaging device, and both imaging modes are combined in time through alternating operation. This merging approach eliminates the need for separate cameras while preserving the functionality of both imaging types
2Adaptability or versatility
If separate cameras are used for visible and NIR imaging, then imaging functionality is improved, but spatial resolution is degraded
Solution Approach 1:
By making the visible camera multi-functional, the system achieves both visible and NIR imaging with a single high-resolution sensor. This avoids the spatial resolution degradation that would occur if a separate lower-resolution NIR camera were used, while still providing versatile imaging functionality across both spectral ranges
3Speed
If mode switching is implemented for NIR and visible imaging, then real-time visualization is improved, but time efficiency is degraded
Solution Approach 1:
The system uses periodic action by alternating between visible mode and NIR mode in rapid succession. The camera switches between capturing visible light and NIR fluorescence images at regular intervals, creating a time-multiplexed operation. This periodic switching allows both imaging modes to function in real-time without requiring permanent separate systems, while the fast switching minimizes the time loss between modes
4Device complexity
If a single camera is used for both visible and NIR imaging, then device complexity is reduced, but imaging functionality is degraded
Solution Approach 1:
The system applies dynamics by making the camera's operational mode changeable over time. The camera dynamically switches between visible-range imaging mode and NIR fluorescence mode based on surgical needs. This dynamic reconfigurability allows a single static device to provide multiple functional modes, reducing device complexity while maintaining full imaging versatility
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Enables continuous, real-time visualization of surgical sites with NIR fluorescence on approximate full-color images, improving surgical efficiency by eliminating the need for mode switching and preserving image quality.
Implementation Method 1
the third sensor configured to capture the surgical scene as illuminated by a near-infrared (NIR) light source
Data Source
AI summary
The technology described herein can be embodied in a method that includes receiving data representing information captured using a first sensor and a second sensor of a multi-sensor camera. The sensors are configured to capture a surgical scene illuminated by a light source configured to emit wavelengths in the visible spectrum corresponding to sensing capabilities of the first and second sensors, respectively. The method also includes receiving data representing information captured using a third sensor of the multi-sensor camera, the third sensor configured to capture the surgical scene as illuminated by a near-infrared light source, and generating a first visual representation of the surgical scene based on the data representing the information captured using the first and second sensors. The first visual representation is combined with the information captured using the third sensor to generate a second visual representation, and the second visual representation is presented on a display device.