3D Telestration Spatial Memory for Surgical Scene Annotations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing telestration technologies in computer-assisted surgical systems fail to integrate 2D drawn objects seamlessly into 3D images, leading to confusion and difficulty in conveying multiple telestration elements during surgical procedures.
Innovation Solution
A system and method that utilizes depth data to render telestration elements in 3D, recording their position and providing spatial memory to ensure they appear in physical contact with the surface, and allows for their subsequent recall after removal.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If telestration elements are drawn on 2D images to communicate with surgeons, then communication between surgical team members and surgeons is enabled, but the drawn objects appear floating and not integrated into the 3D image
Solution Approach 1:
The patent transforms 2D telestration drawings into 3D spatially-accurate annotations by mapping 2D screen coordinates to 3D world coordinates using depth map data. This dimensionality change allows annotations to be placed precisely on 3D surgical scene elements rather than floating on a 2D plane, resolving the spatial integration issue while maintaining communication effectiveness.
Solution Approach 2:
The system introduces depth map data as an intermediary between the 2D drawing interface and the 3D surgical scene. This intermediary layer enables accurate translation of 2D drawing actions into 3D spatial positions, allowing team members to draw on 2D displays while achieving precise 3D annotation placement without requiring the surgeon to switch between 2D and 3D views.
2Loss of information
If multiple telestration elements are drawn during a surgical session to convey information, then more information can be communicated, but it becomes difficult for the surgeon to remember all the information
Solution Approach 1:
The system implements feedback by providing surgeons with the ability to recall previously drawn telestration elements through a simple interface action. When a team member draws an annotation, it is stored and can be retrieved later by the surgeon with a single click, ensuring that important information is not lost and can be reviewed at appropriate moments during the procedure without increasing cognitive load.
Solution Approach 2:
The system performs preliminary action by automatically storing and preserving all telestration elements as they are created, rather than requiring the surgeon to actively remember them. This automatic preservation happens in the background, and the surgeon can access these pre-stored annotations when needed, eliminating the need for active memory maintenance while preserving all communicated information.
3Ease of manufacture
If 2D drawings are used for telestration, then implementation is simple, but the drawn objects do not appear integrated into the 3D surgical scene
Solution Approach 1:
The system maintains implementation simplicity by allowing team members to draw on 2D displays, but automatically performs dimensionality transformation using depth map data to place these drawings in the correct 3D positions. This automatic 2D-to-3D mapping achieves accurate visual integration without requiring complex manual 3D positioning or changing the simple drawing interface.
Solution Approach 2:
The system implements self-service by automatically calculating and applying the correct 3D positioning transformations without requiring manual intervention from the drawing user. The depth map data and coordinate transformation algorithms work autonomously to integrate 2D drawings into the 3D scene, maintaining ease of use while achieving reliable visual integration.
Data Source
AI summary
An exemplary system is configured to detect user input directing a telestration element to be drawn within an image depicting a surface within a scene; render, based on depth data representative of a depth map for the scene and within a three dimensional (3D) image depicting the surface within the scene, the telestration element; record a 3D position within the scene at which the telestration element is rendered within the 3D image; detect a telestration termination event that removes the telestration element from being rendered within the 3D image; and indicate, subsequent to the telestration termination event, an option to again render the telestration element at the 3D position.