Video See-Through AR Rendering With Camera Confidence Maps

The patent captures real-world scenes using cameras and creates virtual views that replicate the physical environment. Instead of relying on complex optical see-through mechanisms, the system copies the real-world visual information through image capture and processing, displaying it on displays within the head-mounted device. This approach eliminates the need for complex optical elements like waveguides and projectors while expanding the field of view beyond physical limitations.

The patent replaces the mechanical and optical system of OST AR (which requires projectors, waveguides, and complex optical elements) with a video-based system using cameras, processors, and displays. This substitution of the optical-mechanical pipeline with an electronic video processing system simplifies the device architecture while achieving wider fields of view and greater usage flexibility.

The patent performs preliminary actions by capturing images with multiple cameras and pre-processing them to determine contributions and generate confidence maps before final virtual view synthesis. This advance preparation of image data from multiple cameras allows the system to efficiently combine information later, maintaining high contextual AR quality while managing computational complexity through staged processing.

The patent introduces confidence maps as intermediary data structures that mediate between multiple camera inputs and the final virtual view synthesis. These confidence maps provide a systematic way to evaluate and weight contributions from different cameras, enabling the processor to intelligently combine multiple image sources while managing computational resources efficiently.