Virtual Pointing Device for Endoscopic Video Using Voice and Gesture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Minimally invasive surgical procedures face challenges in directing attention to tissue anomalies and interacting with equipment due to the indirect view and hands being occupied, necessitating innovative solutions for precise communication and instrument control.
Innovation Solution
The Virtual Pointing Device (VPD) software tool uses audio keywords and hand movements to overlay synthetic visual cues on endoscopic video, combined with edge detection and AI for precise instrument direction, while Cloudcapture provides a scalable infrastructure for surgical video management and sharing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If hand gestures are used to point at tissue anomalies, then communication precision is improved, but hands are occupied and cannot control surgical instruments
Solution Approach 1:
The system introduces voice commands as an intermediary control method. The voice recognition module captures verbal instructions from the surgeon and translates them into system commands, allowing the surgeon to control the virtual pointer and other functions without using hand gestures, thus resolving the conflict between communication precision and instrument control
Solution Approach 2:
The system replaces manual hand gesture control with voice-based acoustic control. The voice recognition module converts spoken commands into digital signals that control the virtual pointer's position and function, substituting the mechanical hand gesture system with an acoustic control system that does not interfere with surgical instrument manipulation
2Ease of operation
If voice commands are used to control the system, then hands remain free for instrument manipulation, but voice recognition accuracy may be affected by surgical environment noise
Solution Approach 1:
The system implements feedback mechanisms where the recognized voice commands are displayed or confirmed to the surgeon, allowing for verification and correction. The system can request clarification or repeat commands when recognition confidence is low, adapting to the noisy surgical environment through iterative feedback loops that improve recognition accuracy
3Loss of information
If virtual pointer functionality is added to the system, then communication capability is improved, but device complexity increases
Solution Approach 1:
The virtual pointer system is designed to work with multiple types of surgical instruments and endoscopic views through a unified interface. The same voice commands and control mechanisms work across different surgical contexts, reducing the need for separate specialized functions and thereby managing complexity while maintaining versatility in information communication
Data Source
AI summary
The present invention is a foundational video architecture and a framework for collecting surgical video at an enterprise scale. It enables hospitals to ingest, manage, and fully utilize patient surgical video within the hospital and to share video with designated users outside of the hospital. It is a passive solution that automatically records video during surgery and provides key clips post-surgery. It stores video to the cloud and integrates videos to the patient's Electronic Health Record (EHR). The present invention pertains to the need to precisely identify and communicate a specific position on tissue or an organ with a Virtual Pointing Device (VPD) software tool. The VPD uses a combination of audio key words and the surgeon's hand movements to invoke various functionality.