Virtual Object Realism in Mixed Reality Teaching via Depth and Lighting Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current mixed reality (MR) technologies struggle to generate a realistic sense of virtual objects in teaching scenes, as virtual objects lack understanding of the real environment, often penetrate other objects, and lighting and shadow effects are not convincingly rendered, leading to a lack of immersion and interaction realism.
Innovation Solution
A method involving depth data collection, 3D model generation using semantic segmentation, real-time scene perception, adaptive lighting and shadow effects, and multi-modal interactive prompts to enhance the realism and interaction of virtual objects in an augmented teaching environment, utilizing techniques like ShadowMap, Raycasting, and custom Shaders for optimized rendering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If virtual objects are placed in augmented teaching scenes using MR technology, then interactivity and immersion are improved, but the virtual objects lack understanding of the real environment causing them to penetrate objects and lack realism
Solution Approach 1:
The system performs preliminary actions by collecting depth data of the teaching space before placing virtual objects, constructing 3D models and occlusion maps in advance. This allows the virtual objects to understand the real environment structure beforehand, preventing penetration issues and enhancing realism while maintaining interactivity.
2Ease of operation
If simple virtual object placement is used, then ease of operation is improved, but lighting effects and shadow effects are not real resulting in lack of immersion
Solution Approach 1:
The system implements self-service by automatically collecting lighting information from the teaching environment and generating appropriate lighting effects and shadow maps for virtual objects without requiring manual configuration. This maintains ease of operation while achieving realistic lighting and shadow effects that enhance immersion.
3Loss of information
If depth data collection from multiple trajectories is performed, then understanding of teaching scene is improved, but collection time and processing complexity increase
Solution Approach 1:
The system applies segmentation by dividing the teaching space into multiple trajectories and collecting depth data from different angles separately. This allows comprehensive scene understanding while organizing data collection in a structured manner that reduces processing complexity and time compared to single-trajectory approaches.
4Reliability
If occlusion relationships are determined for all objects, then realism is improved, but computational complexity increases making real-time rendering difficult
Solution Approach 1:
The system implements local quality by determining occlusion relationships selectively for objects within the field of view and those that actually occlude or are occluded by virtual objects. This maintains high occlusion accuracy for relevant objects while reducing computational complexity by avoiding unnecessary calculations for all objects in the scene.
Data Source
AI summary
A method for generating sense of reality of a virtual object in a teaching scene includes perception of a teaching space, generation of sense of reality of a virtual object and generation of real effect of dynamic interaction. The method is specifically performed through steps of collecting depth data of the teaching space; perceiving changes of a scene object in a field of view in real time; collecting a light intensity in the teaching scene to realize a virtual-real fused lighting effect; generating a shadow effect of the virtual object in real time by using ShadowMap; and guiding a teacher to use a multi-modal algorithm to complete a real-time interaction with the virtual object by setting interactive prompts of a sight target and a virtual hand.