Non Photorealistic Rendering for AR Depth Perception
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Medical augmented reality displays face challenges in providing accurate depth perception due to inaccurate occlusion handling, particularly in complex surgical environments like cardiothoracic and gastrointestinal surgery, where virtual objects appear to float above the tissue surface despite being rendered at correct depths.
Innovation Solution
A method involving non-photorealistic rendering (NPR) of images to create a partly transparent window based on captured image data, which allows for occlusion cues without requiring a 3D model of the underlying scene, by assigning transparency values based on image intensity gradients and saliency maps to enhance depth perception.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Illumination intensity
If virtual objects are rendered opaquely in AR, then they are clearly visible, but depth perception deteriorates as objects appear to float above the tissue surface
Solution Approach 1:
The patent applies different transparency properties to different regions of the anatomical surface. Salient features (ridges, folds, boundaries) are rendered with higher opacity to maintain visibility and depth cues, while non-salient regions are rendered with lower opacity to allow visibility of virtual objects behind them. This local differentiation resolves the contradiction by making the surface selectively transparent rather than uniformly so.
Solution Approach 2:
The patent modifies the visual properties of the anatomical surface by applying transparency values based on saliency detection. The rendering system changes the optical properties (transparency/opacity) of different surface regions to provide depth cues while maintaining visibility of both the surface features and underlying virtual objects.
2Measurement precision
If the anatomical surface is rendered transparent to show virtual objects, then depth perception improves, but occlusion cues are lost making it difficult to distinguish salient features
Solution Approach 1:
The patent computes a saliency map identifying important anatomical features and applies selective transparency: salient features maintain high opacity to preserve occlusion cues and depth information, while non-salient areas use low opacity to reveal virtual objects. This local quality approach prevents loss of critical information while enabling depth perception.
Solution Approach 2:
The anatomical surface is segmented into salient and non-salient regions based on gradient analysis and feature detection. This segmentation allows the system to apply different transparency treatments to different segments, preserving occlusion cues where needed while providing depth perception where appropriate.
3Measurement precision
If a window is created in the anatomical surface to show virtual objects, then depth perception improves, but information about the body surface within the window is lost
Solution Approach 1:
Instead of creating a uniform window that removes surface information, the patent applies local quality modulation: the saliency-based transparency system preserves surface information in salient regions while selectively transparentizing non-salient regions. This maintains body surface information while providing depth perception cues.
Data Source
AI summary
A method and system for rendering a captured image of a scene is disclosed which provide see-through vision through the rendered image, for example of an augmented reality object rendered behind the scene, by assigning pixel transparency values in dependence upon captured image pixels. The method and system preserve some structure of the scene in the rendered image without requiring a model of the scene.