See-Through Console Overlay for Minimally Invasive Surgery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current minimally invasive surgical systems, such as the da Vinci Surgical System, rely heavily on oral communications for instruction, which are limited in expressiveness and clarity, especially when compared to natural hand gestures, and may not be practical in all surgical environments due to requirements for constant color backgrounds or open surgery settings.
Innovation Solution
The system overlays real-time hand gestures onto a surgical site image using depth data to provide clearer instructions, allowing hand gestures to communicate tasks like grasping, cutting, or manipulating instruments, and enables a see-through console display by segmenting and combining images of the surgeon's hands and master manipulators with the surgical site, using alpha masks generated from depth data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If oral communications are used for instruction, then the system is simple to operate, but the expressiveness and clarity of instruction is limited
Solution Approach 1:
The system captures and reproduces hand gestures visually by segmenting the instructor's hand from the video background and overlaying it on the surgical site display. This creates a visual copy of the gesture that preserves the full expressiveness of natural hand movements while maintaining system simplicity
Solution Approach 2:
The system transitions from one-dimensional oral communication to two-dimensional visual display by projecting hand gestures onto the surgical site image plane, adding spatial and visual dimensions to the instruction modality
2Ease of manufacture
If constant color background is used for hand segmentation, then hand gesture extraction is simplified, but the applicability to real surgical environments is reduced
Solution Approach 1:
The system changes the segmentation parameter from color-based (requiring constant color background) to depth-based segmentation using time-of-flight data. This allows hand gesture extraction in real surgical environments with varying colors and lighting conditions
Solution Approach 2:
The system replaces the mechanical requirement of a constant color background with an optical sensing approach using time-of-flight depth sensing, eliminating the need for controlled environmental conditions
3Loss of information
If the console display shows only the surgical site, then the display is simple and clear, but the surgeon cannot see their hands and manipulators simultaneously
Solution Approach 1:
The system merges multiple visual elements (surgical site image, hand image, manipulator image) into a single composite display by overlaying segmented hand and manipulator images on the surgical site, providing complete visual information in one view
Solution Approach 2:
The system segments different visual elements (hand, manipulator, surgical site) from their respective sources and selectively combines them, allowing the surgeon to see all elements simultaneously without visual interference
Data Source
AI summary
In a minimally invasive surgical system, a plurality of video images is acquired. Each video image includes images of the surgeon's hand(s), and of a master manipulator. The images of the surgeon's hand(s) and the master manipulator are segmented from the video image. The segmented images are combined with an acquired surgical site image. The combined image is displayed to the person at the surgeon's console so that the console functions as a see-through console.