Machine-Learning Visual-Haptic Feedback for Robotic Surgery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Robotic surgical platforms lack direct physical haptic feedback, relying on visual haptics that are inconsistent and inaccurate, varying among surgeons of different skill levels.
Innovation Solution
A machine-learning-based visual-haptic feedback system that analyzes endoscopic videos to predict surgical tool-tissue interaction strength levels, converting these into physical feedback signals for surgeons via user interfaces.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If visual haptics are used to gauge applied forces in robotic surgery, then surgeons can operate without physical haptic feedback, but the accuracy and consistency of force judgment varies significantly among surgeons of different skill levels
Solution Approach 1:
The patent replaces the mechanical haptic feedback system with a computational vision-based system. Instead of relying on surgeons' visual interpretation of tissue deformation, the system uses deep learning models to automatically quantify applied forces from endoscopic video images, substituting mechanical sensing with computational analysis to achieve objective, consistent force measurement.
Solution Approach 2:
The patent introduces an intermediary layer between the surgical tool-tissue interaction and the surgeon's perception. The deep learning model acts as a mediator that processes visual information from the endoscopic camera and converts it into quantitative force estimates, providing a standardized bridge between visual observations and physical force measurements.
2Adaptability or versatility
If visual haptics are used to assess surgical forces, then robotic surgical platforms can be operated, but there is no good way to provide a consistent correlation between visual appearances and applied forces
Solution Approach 1:
The patent replaces the unreliable visual correlation method with a computational model that directly quantifies forces. The deep learning architecture processes endoscopic images and outputs objective force measurements, eliminating the subjectivity and inconsistency inherent in visual haptic interpretation while maintaining adaptability to various surgical scenarios.
Solution Approach 2:
The patent implements a feedback mechanism where the system continuously monitors the surgical tool-tissue interaction through endoscopic video, processes the visual information through the deep learning model, and provides real-time force estimates to the surgeon. This closed-loop feedback system ensures consistent and reliable force correlation throughout the surgical procedure.
3Measurement precision
If more experienced surgeons use visual haptics, then they can make more accurate mappings, but there is still no standardized way to ensure consistent force assessment across all surgeons
Solution Approach 1:
The patent replaces the skill-dependent visual haptic method with an automated computational system. The deep learning model serves as an objective, experience-independent arbiter that consistently quantifies applied forces from visual data, eliminating the variability introduced by different surgeon expertise levels while maintaining high measurement precision.
Solution Approach 2:
The system performs self-service by automatically analyzing the surgical tool-tissue interaction without requiring surgeon expertise for force assessment. The deep learning model independently processes visual information and provides accurate force estimates, freeing the surgeon from the burden of interpreting visual cues based on their experience level.
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
Embodiments described herein provide various examples of a visual-haptic feedback system for generating a haptic feedback signal based on captured endoscopy images. In one aspect, the process for generating the haptic feedback signal includes the steps of: receiving an endoscopic video captured for a surgical procedure performed on a robotic surgical system; detecting a surgical task in the endoscopic video involving a given type of surgical tool-tissue interaction; selecting a machine learning model constructed for analyzing the given type of surgical tool-tissue interaction; for a video image associated with the detected surgical task depicting the given type of surgical tool-tissue interaction, applying the selected machine learning model to the video image to predict a strength level of the depicted surgical tool-tissue interaction; and then providing the predicted strength level to a surgeon performing the surgical task as a haptic feedback signal for the given type of surgical tool-tissue interaction.