Mixed Reality Surgical Visualization for Off-Screen Object Rendering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing surgical procedures lack an effective augmented reality (AR) interactive experience that enhances real-world surgical environments with computer-generated sensory information, such as visual, auditory, haptic, and olfactory feedback, to improve situational awareness and efficiency.
Innovation Solution
A method and system for mixed reality visualization in surgical systems that overlay computer-generated sensory information, including visual, auditory, haptic, and olfactory feedback, onto real-world surgical environments using AR devices and imaging systems, allowing selective display management based on device type, risk, and surgeon experience.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple sensory modalities (visual, auditory, haptic, olfactory) are overlaid onto real-world surgical images, then the augmented reality interactive experience is enhanced, but the device complexity and information processing requirements increase
Solution Approach 1:
The system segments the augmented reality experience into separate sensory modalities (visual overlays, auditory feedback, haptic feedback, olfactory feedback) that can be independently controlled and managed. Each sensory channel is processed and delivered through dedicated components, allowing the system to provide enhanced AR experiences while managing complexity through modular architecture.
Solution Approach 2:
The surgical system is designed with multi-functional capabilities that integrate multiple sensory modalities into a unified AR platform. The system can selectively activate different sensory channels based on surgical needs, making it adaptable to various surgical scenarios while sharing common infrastructure for image processing and display management.
2Loss of information
If real-time image streaming and overlay processing are implemented, then situational awareness is improved, but the processing time and computational resources increase
Solution Approach 1:
The system performs preliminary processing of surgical images and preparation of augmented reality overlays before they are needed during the surgical procedure. Pre-computed anatomical models, pre-loaded surgical pathways, and pre-processed imaging data reduce the computational burden during real-time execution, enabling rapid display updates without compromising situational awareness.
Solution Approach 2:
The system maintains continuous real-time image streaming and overlay rendering without interruption throughout the surgical procedure. By establishing persistent data flows and continuous rendering pipelines, the system ensures that situational awareness information is always available with minimal latency, avoiding the need for repeated processing cycles.
3Loss of information
If selective display management based on device type, risk, and surgeon experience is implemented, then the relevance of AR information is improved, but the control system complexity increases
Solution Approach 1:
The system applies selective display management by tailoring the AR information presentation to the specific context, surgeon experience level, and device capabilities. Different surgical teams receive customized information streams with varying levels of detail and complexity, ensuring that each user receives information optimally suited to their needs without requiring a completely separate system for each configuration.
Solution Approach 2:
The control system dynamically adjusts the AR information display based on real-time conditions including surgeon preferences, surgical phase, and device type. The system can switch between different display configurations and information hierarchies during the procedure, allowing flexible adaptation to changing requirements without permanent system reconfiguration.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Apparatuses, systems, and methods for method for mixed reality visualization are disclosed herein. In one aspect, a method for mixed reality visualization includes capturing, by a first camera of a first visualization system, an image of an object in a surgical field, wherein a first portion of the object is outside of a field of view of the first camera; tracking, by a tracking system, a position of a second portion of the object; determining, by a surgical hub, an attribute of the object based on the tracked position of the second portion of the object, wherein the attribute of the object is related to the first portion of the object outside of a field of view of the camera; and displaying, by an augmented reality display device, the captured image of the object in the surgical field and a graphic based on the attribute of the object.