Secondary-Camera Ghost Visualization for Obscured AR Surgery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
AR devices have a limited field of view that can be obscured by a user's hands or instruments, leading to disorientation and fatigue during surgical procedures.
Innovation Solution
A system with multiple cameras and a controller that determines the position and perimeter of obscuring objects, using computer vision algorithms to stitch together feeds and display augmented reality information from a second camera inside a wireframe, maintaining visibility of obscured objects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If a single field of view is used in AR devices, then the device complexity is reduced, but the visibility of surgical objects is lost when obscured by hands or instruments
Solution Approach 1:
The system divides the visual information capture into multiple independent camera units, each with its own field of view. The first camera captures the primary surgical scene while the second camera captures obscured areas, allowing the system to segment the monitoring task across multiple sensors to maintain continuous visibility.
Solution Approach 2:
The processor acts as an intermediary that receives feeds from multiple cameras, determines obscuration events, and synthesizes a composite view. It mediates between the limited field of view of individual cameras and the requirement for complete visibility by selectively combining feeds and displaying augmented reality information from the second camera when obscuration is detected.
2Loss of information
If multiple cameras are used to maintain visibility, then the visibility of obscured objects is improved, but the device complexity increases
Solution Approach 1:
The system dynamically switches between different camera feeds based on real-time obscuration detection. The processor continuously monitors the first camera feed, detects when obscuration occurs, and dynamically integrates information from the second camera feed, creating a dynamic adaptation to changing surgical conditions rather than a static multi-camera setup.
Solution Approach 2:
The system applies different processing and display strategies to different regions of the visual field. When obscuration is detected in specific areas, the processor selectively augments those local regions with information from the second camera, rather than uniformly processing all visual information, thereby optimizing the use of multiple camera feeds.
3Loss of information
If the field of view is expanded to avoid obscuration, then the visibility is improved, but the AR device may obscure the real-world view
Solution Approach 1:
Instead of expanding the horizontal or vertical field of view, the system adds a temporal dimension by capturing and processing visual information from multiple time points and angles. The second camera captures obscured areas that are then integrated into the augmented reality display, effectively viewing the scene from another spatial dimension through multiple camera positions rather than expanding a single camera's view.
Data Source
AI summary
Described are methods and systems for a viewing system, comprising an augmented reality (AR) system having a first camera with a first point of view, the AR system having an AR display configured to display augmented reality information overlaid over a real world scene, the first point of view of the first camera having a same point of view as the AR display, and the scene containing a first physical object in a field of view of the first camera; a second camera having a second point of view different from the first point of view, wherein the first physical object is in the field of view of the second camera; and a controller configured to: receive an input that the first physical object in the real world scene has become obscured by a second physical object from the first point of view of the first camera; determine a position of the second object that has obscured the first object; determine a perimeter of the second object; display augmented reality information representing the determined perimeter of the second object; extract images from the second camera of the first physical object; and display the extracted images as augmented reality information inside the perimeter of the second object, such that the first object appears to be visible through the second object.