Deviceless AR System Using Depth Cameras and Projectors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current augmented reality and interaction technologies require cumbersome head-mounted displays and tracking devices to create immersive 3D environments, limiting their effectiveness in providing seamless interactions beyond the confines of a display.
Innovation Solution
A system combining multiple depth-sensing cameras and projectors to create a unified 3D model of a space, allowing for interactive experiences by projecting graphics onto objects and users, enabling interactions in a volume rather than just on surfaces, and using the environment and user body as interactive displays.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If head-mounted displays and tracking devices are used to create immersive 3D environments, then the immersive experience is improved, but the device complexity and user burden increase
Solution Approach 1:
The system divides the immersive environment into multiple independent projection zones and depth-sensing regions, allowing the space itself to be segmented into interactive areas without requiring unified head-mounted displays. Each region can be independently controlled and sensed, reducing the need for complex integrated tracking devices.
Solution Approach 2:
The patent transitions from 2D display surfaces to 3D volumetric interaction space by using depth cameras and spatial projection. This dimensional expansion allows users to interact with virtual objects in mid-air and throughout the room volume, eliminating the need for head-mounted displays while maintaining immersion through spatial awareness.
2Device complexity
If interactions are confined to the physical extent of the display, then the system simplicity is maintained, but the interaction space is limited
Solution Approach 1:
The system makes any surface in the environment (walls, tables, floors, user bodies) function as an interactive display surface through projection and depth sensing. This universal interaction capability allows the entire room to serve as the interaction space, eliminating the need for dedicated display hardware while expanding the volumetric interaction region.
Solution Approach 2:
The environment itself serves as the interaction interface by using naturally present surfaces (walls, furniture, user bodies) as projection screens and sensing targets. The system leverages the existing physical space without requiring additional specialized equipment, allowing the environment to serve its own interactive function.
3Volume of moving object
If multiple depth cameras and projectors are used to cover a specified space, then the interaction volume is expanded, but the device complexity increases
Solution Approach 1:
The patent integrates multiple depth cameras and projectors into a unified coordinate system through calibration, merging their individual functions into a cohesive spatial interaction system. This consolidation allows the multiple devices to work together as a single integrated environment rather than separate components, managing complexity through unified control.
Solution Approach 2:
The calibrated coordinate system acts as an intermediary that translates between the different camera and projector reference frames, enabling seamless integration of multiple devices. This mathematical mediation allows the system to handle multiple sensors and projectors without requiring complex direct coordination between each component.
4Loss of information
If the user's body is used as a temporary canvas for projection, then the visual continuity is improved, but the measurement precision requirements increase
Solution Approach 1:
The system performs preliminary calibration of the depth cameras and projectors to establish accurate coordinate mappings before actual interaction begins. This pre-calibration ensures that when the user's body is used as a projection surface, the spatial relationships and measurement precision are already optimized, reducing the precision burden during dynamic interaction.
Data Source
AI summary
Architecture that combines multiple depth cameras and multiple projectors to cover a specified space (e.g., a room). The cameras and projectors are calibrated, allowing the development of a multi-dimensional (e.g., 3D) model of the objects in the space, as well as the ability to project graphics in a controlled fashion on the same objects. The architecture incorporates the depth data from all depth cameras, as well as color information, into a unified multi-dimensional model in combination with calibrated projectors. In order to provide visual continuity when transferring objects between different locations in the space, the user's body can provide a canvas on which to project this interaction. As the user moves body parts in the space, without any other object, the body parts can serve as temporary “screens” for “in-transit” data.