Virtual Slate AR Interface for Surgical Gesture Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional 3D visualization systems lack the ability to allow users to interact and manipulate rendered objects through physical gestures, limiting user engagement and interaction in augmented reality environments.
Innovation Solution
The Interaction Engine tracks user movements in the physical world and translates them into virtual interactions within a unified 3D coordinate space, enabling the manipulation and selection of virtual objects through gestures, such as those facilitated by an AR virtual touchpad or trackpad, and adjusting visual data displays based on user input.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If conventional 3D visualization systems are used, then the system structure is simple, but the user interaction capability is limited
Solution Approach 1:
The patent introduces a virtual slate as an intermediary interface between the user and the 3D visualization system. This virtual slate, displayed within the augmented reality environment, serves as a touch-sensitive control surface that mediates user gestures and translates them into system commands, thereby enhancing interaction capability without requiring direct modification of the core visualization system architecture
Solution Approach 2:
The patent replaces traditional mechanical input devices (physical touchpads, keyboards, or mice) with a virtual projection system. The virtual slate is rendered optically within the AR field of view, and user interactions are detected through optical sensors or cameras tracking hand movements, substituting mechanical contact with optical detection and virtual feedback
2Adaptability or versatility
If physical gesture interaction is added to 3D visualization systems, then user engagement improves, but system complexity increases
Solution Approach 1:
The virtual slate is designed as a universal interface that can perform multiple functions including navigation, parameter adjustment, data selection, and control commands. This single multi-functional virtual interface replaces the need for multiple specialized input devices or complex gesture recognition systems, thereby improving adaptability while controlling overall system complexity
Solution Approach 2:
The patent adds a new dimension of interaction by projecting the control interface into the third spatial dimension within the AR environment. Instead of requiring users to look at and touch a separate physical device, the virtual slate appears in the same spatial space as the visualized data, allowing intuitive hand gestures to interact with 3D objects from any viewing angle, thereby enhancing user engagement through spatial immersion
Data Source
AI summary
Various embodiments of an apparatus, methods, systems and computer program products described herein are directed to an Interaction Engine. According to various embodiments, the Interaction Engine generates, within a unified three-dimensional (3D) coordinate space: a virtual 3D medical model positioned according to a current model pose and at least one a virtual 3D hand representation. The Interaction Engine renders an AR display that includes display of the virtual 3D medical model positioned according to the current model pose and the virtual 3D hand representation. The Interaction Engine identifies selection of a slate virtual interaction based on the type of movement of the virtual 3D hand representation. The Interaction Engine modifies the AR display by rendering a virtual slate concurrently displayed with the virtual 3D medical model. The Interaction Engine adjusts the display of the virtual 3D medical model based at least on subsequent physical gestures with respect to the virtual slate.