VR HMI Prototyping With 2D-3D Interface Integration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing prototyping methods for products, particularly those involving human-machine interfaces, struggle with integrating 2D-designed screen interfaces into 3D environments and physical components, lacking ease in verifying real-world functionality and requiring costly and lengthy physical mock-ups.
Innovation Solution
A system and method that seamlessly integrate 2D design software with 3D graphics engines, using a master device and slave devices with actuators and sensors, enabling interactive prototyping through virtual reality or augmented reality headsets, allowing designers to test 2D HMI interfaces within 3D virtual models and physical mock-ups.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If physical mock-ups are built for prototyping, then realistic user experience can be reproduced, but the process becomes lengthy and costly
Solution Approach 1:
The patent creates virtual copies (3D digital models) of physical products that can be interacted with in VR environments. These digital twins replicate the appearance, structure, and behavior of physical mock-ups without requiring physical materials, thereby reducing both time and cost while maintaining verification reliability
Solution Approach 2:
The patent replaces physical mechanical mock-ups with virtual reality-based digital models. By substituting physical construction with software rendering and virtual interaction, the system eliminates the time-consuming physical prototyping process while preserving the ability to verify user experience through immersive VR visualization
2Ease of manufacture
If 2D interface design software is used separately, then interface design is straightforward, but verification in real-world scenarios is difficult
Solution Approach 1:
The patent merges 2D interface design capabilities with 3D virtual environment visualization. The system integrates screenshot extraction from 2D interface designs and embeds them into 3D VR scenes, allowing designers to verify how interfaces appear and function in realistic contextual scenarios while maintaining the simplicity of 2D design tools
Solution Approach 2:
The patent introduces screenshot images as an intermediary between 2D interface design and 3D environmental verification. By converting interface elements into image formats that can be embedded in VR scenes, the system bridges the gap between simple 2D design tools and realistic scenario verification without requiring complex 3D modeling of interfaces
3Reliability
If 3D VR/AR prototyping tools are used, then immersive visualization is achieved, but designing 2D interfaces within them is complex or impossible
Solution Approach 1:
The patent segments the interface design process from the environmental modeling process. Instead of attempting to create full 3D models of interfaces within VR tools, the system extracts 2D screenshots from separate interface design software and embeds them as image elements into the 3D VR scene, simplifying both tasks while maintaining immersive visualization
4Loss of substance
If virtual reality prototyping is used, then cost is reduced, but integration with physical components is limited
Solution Approach 1:
The patent creates a hybrid prototyping system that serves multiple functions: it can visualize purely virtual scenarios using 3D models and embedded screenshots, and it can also integrate with physical mock-ups through VR/AR headsets. This multi-functionality allows the same virtual prototype to be used for both digital-only verification and physical-hardware integration testing, enhancing versatility while maintaining cost efficiency
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A system (1) for the interactive prototyping of products includes a computer (40), a VR or AR headset (HS) wearable by a user, a master device (10) and a slave device (20). The computer runs a 2D design software program (PT) for designing a 2D screen interface that is intended to be incorporated into a product, runs a 3D graphics engine (GE) for producing a 3D model of the product, and incorporates the 2D screen interface into the 3D model of the product to produce a virtual representation of the product. The headset (HS) is coupled to the computer and displays the virtual representation of the product. The master device is coupled to the computer (40) via one or more serial communication interfaces (12a, 12b) to exchange data between the 2D design software program (PT), the 3D graphics engine (GE) and the master device (10) via a software plugin (PL1, PL2) run by the computer (40). The master device (10) includes a first wireless communication interface (14). The slave device (20) includes a second wireless communication interface (24) for exchanging data with the first wireless communication interface (14), and further includes an actuator (25) and/or a sensor (27, 28, 29). Actuation data is transmitted from the 2D design software program (PT) and/or the 3D graphics engine (GE) to the actuator (25) via the master device (10), and control data is transmitted from the sensor (27, 28, 29) to the 2D design software program (PT) and/or the 3D graphics engine (GE) via the master device (10).