Virtual Pointer for Endoscopic Video Using Voice and Gesture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Minimally invasive surgical procedures face challenges in directing and focusing viewer attention due to the lack of intuitive means for surgeons to point to tissue anomalies or physical artifacts, as hand gestures are not easily replicable in indirect views, hindering communication and interaction with supporting staff.
Innovation Solution
A Virtual Pointing Device (VPD) software tool that uses a combination of audio keywords and hand movements to overlay a synthetic visual line from the surgical instrument, analyzing live endoscopic video to determine direction using edge detection algorithms, artificial intelligence, or neural networks, allowing precise communication of instrument position.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If traditional hand gestures are used to point to tissue anomalies, then communication is intuitive and direct, but this method is not feasible in minimally invasive surgery where the surgeon's hands are occupied with instruments
Solution Approach 1:
The system introduces an intermediary computational layer that translates the surgeon's instrument movements and voice commands into virtual pointing indicators. The voice command system acts as a mediator, allowing the surgeon to issue pointing commands without using hand gestures, while the system processes the audio input and generates the appropriate visual indicators on the video feed.
Solution Approach 2:
The patent replaces the mechanical hand gesture system with a voice-based control system. Instead of using physical hand movements to indicate direction, the surgeon uses voice commands combined with instrument orientation data. The system substitutes the mechanical gesture interface with an audio interface and computational processing, enabling pointing functionality without requiring free hands.
2Ease of operation
If voice commands are used to control surgical instruments, then hands remain free for manipulation, but precise directional indication becomes more complex
Solution Approach 1:
The system merges multiple functions into a unified voice command interface. Instead of requiring separate controls for instrument manipulation and directional indication, the system combines voice recognition with instrument tracking algorithms. The voice commands work in conjunction with automated computer vision systems that detect instrument orientation and position, merging manual voice control with automated visual analysis to achieve precise directional indication.
Solution Approach 2:
The voice command system serves multiple functions simultaneously: it controls instrument manipulation, provides directional indication, and communicates with supporting staff. The same voice interface that allows hand-free instrument control also generates the virtual pointing indicators, making the system multi-functional without requiring additional separate controls.
3Measurement precision
If computational algorithms are used to determine instrument direction, then precise positioning is achieved, but processing time and computational resources increase
Solution Approach 1:
The system performs preliminary actions by continuously tracking instrument position and orientation in real-time, even before a voice command is issued. The computer vision system pre-processes video frames to identify instrument characteristics and maintain ready-state directional data. When a voice command is given, the system already has pre-computed instrument orientation information available, reducing the processing delay between command issuance and virtual indicator generation.
Data Source
AI summary
The present invention pertains to the need to precisely identify and communicate a specific position on tissue or an organ with a Virtual Pointing Device (VPD) software tool. The VPD uses a combination of audio key words and the surgeon's hand movements to invoke various functionality.