Mixed Reality Ink Projection Using Spatial Mesh Planes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing mixed reality technologies face challenges in coherently projecting 2D ink strokes drawn by a remote user onto the 3D space of a headset user, particularly in complex environments with noisy or irregular surfaces, leading to incoherent and confusing annotations.
Innovation Solution
A method involving sampling points from the spatial mesh, performing weighted principal component analysis to approximate a projection plane, and projecting the ink stroke onto this plane, which is retained across multiple strokes unless significant time or spatial separation occurs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If 2D ink strokes are projected directly onto the 3D spatial mesh of the headset user's environment, then the annotations can be placed in the mixed reality space, but the annotations become incoherent and confusing when viewed from different perspectives, especially in complex environments with noisy or irregular surfaces
Solution Approach 1:
The patent transforms the 2D ink stroke coordinates from screen space into 3D spatial coordinates by casting rays from the camera perspective through each ink point to intersect with the spatial mesh. This dimensional transformation allows annotations to be placed in three-dimensional space rather than being constrained to a two-dimensional plane, enabling coherent visualization from multiple perspectives.
Solution Approach 2:
The patent introduces an intermediary projection plane that is dynamically determined based on the spatial mesh geometry. Instead of directly projecting ink strokes onto the complex spatial mesh, the system first identifies a suitable planar surface (such as a wall, floor, or object surface) and projects the annotations onto this intermediate plane, ensuring coherence and readability from different viewing angles.
2Measurement precision
If multiple ray casts are performed to determine precise projection planes for each ink stroke, then accurate annotation placement is achieved, but the computational complexity and processing time increase significantly
Solution Approach 1:
The patent combines multiple ink stroke processing operations into a unified workflow. Instead of independently determining projection planes for each ink stroke, the system processes a sequence of ink points together, identifying a common projection plane that accommodates multiple strokes. This merging of operations reduces redundant ray casting and computational overhead.
Solution Approach 2:
The patent performs preliminary identification of the projection plane before finalizing the ink stroke annotations. By determining the appropriate planar surface early in the processing pipeline (based on the spatial mesh and camera perspective), the system avoids repeated computational operations for each individual ink point, thereby reducing overall processing complexity.
3Manufacturing precision
If the projection plane is recalculated for each new ink stroke, then accurate placement is maintained, but the workflow efficiency decreases due to repeated calculations
Solution Approach 1:
The patent implements a periodic recalculation strategy for the projection plane rather than calculating it for every single ink stroke. The system determines the projection plane at specific intervals or when certain conditions are met (such as a significant change in drawing location or perspective), allowing multiple ink strokes to be placed on the same pre-determined plane without repeated calculations, thus improving workflow efficiency while maintaining adequate precision.
Data Source
AI summary
Mixed reality images are inked with strokes and other annotations, permitting a headset, robot, or drone operator to see coherent graphical additions made by a remote user. Some embodiments get an ink stroke in an annotation which includes multiple points in a screen space, select a ray origin point based on at least samples of the annotation points, obtain a spatial mesh representation of at least a portion of an object which is at least partially shown in a mixed reality image, cast a ray from the ray origin point to the spatial mesh representation, thereby determining a ray-mesh intersection point, choose a projection plane, project the annotation onto the projection plane, and configure a display with the mixed reality image including the projected ink stroke or other annotation.