AR Brain Tumor Visualization With Deep Learning Segmentation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Minimally invasive brain tumor surgeries face challenges due to the loss of visual and haptic feedback as surgeons rely on separate 2D displays, leading to increased risks and requiring high expertise, limiting their application to experienced neurosurgeons.

Innovation Solution

An augmented reality system using deep learning models for 3D tumor, vasculature, and white matter tract segmentation, integrated with a head-mounted display, provides comprehensive 3D visualization of brain anatomy, reducing the need to switch perspectives and enhancing surgical precision.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If minimally invasive surgery is used, then surgical safety is improved by reducing infection risk and damage to brain matter, but visual feedback is lost as surgeons cannot directly see the surgical site

Engineering Contradiction:

Improvesurgical safetyVSAvoidvisual feedback

Core Design Contradiction:

ReliabilityVSLoss of information

Solution Approach 1:

The patent introduces an augmented reality head-mounted display as an intermediary device that overlays 3D tumor location data directly onto the surgeon's field of view. This mediator bridges the gap between the minimally invasive surgical approach and the need for visual feedback, allowing surgeons to see through the probe and view preoperative scan data without compromising surgical safety.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent transforms 2D preoperative scan data into 3D visualizations that are overlaid in the surgeon's augmented reality field of view. This dimensional transformation allows surgical information to be presented in three dimensions, providing intuitive spatial understanding of tumor location and boundaries while maintaining the benefits of minimally invasive surgery.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Loss of information

If separate 2D displays are used to show preoperative scans, then tumor location information is provided, but surgeon cognitive load increases due to the need to switch perspectives and translate information

Engineering Contradiction:

Improvetumor location informationVSAvoidcognitive load

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The patent merges the surgeon's direct view of the surgical site with overlaid 3D tumor data from preoperative scans into a single augmented reality display. This combination eliminates the need to switch between separate 2D displays and the surgical field, reducing cognitive load by presenting all necessary information in one unified view.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

Instead of requiring the surgeon to mentally translate 2D scan data into 3D spatial understanding, the patent inverts the approach by directly projecting 3D visualizations of the tumor onto the surgeon's field of view. This reversal of the information translation process significantly reduces the cognitive effort required to interpret tumor location and plan resection.

Inventive Principle:

Principle #13The other way round (Inversion)

3Duration of action of moving object

If minimally invasive surgery is used, then recovery time is reduced, but surgical precision is compromised due to lack of direct visualization

Engineering Contradiction:

Improverecovery timeVSAvoidsurgical precision

Core Design Contradiction:

Duration of action of moving objectVSManufacturing precision

Solution Approach 1:

The patent replaces the traditional mechanical optical system of direct visualization (craniotomy with microscope) with an augmented reality optical system. The head-mounted display uses optical waveguides and holographic projection to deliver surgical information directly to the surgeon's eyes, maintaining surgical precision while enabling minimally invasive approaches that reduce recovery time.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS12482202B2Live surgical aid for brain tumor resection using augmented reality and deep learning

Publication Date: 2025.11.25 PAPAR RAYHAN

AI summary

An augmented reality system and method, comprising: a memory configured to store 3D medical scans comprising an image of a tumor and an angiogram; an output port configured to present a signal for presentation of an augmented reality display to a user; at least one camera, configured to capture images of a physiological object from a perspective; at least one processor, configured to: implement a first neural network trained to automatically segment the tumor; implement a second neural network to segment vasculature in proximity to the tumor; implement a third neural network to recognize a physiological object in the captured images; and generate an augmented reality display of the physiological object, tumor and vasculature based on the captured images, the segmented tumor and the segmented vasculature, compensated for changes in the perspective.