Synthesizing Panoramic Views from Sparse Data

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing interactive video tour systems face limitations in providing realistic navigation through remote locations, especially in urban environments, due to the need for a dense grid of panoramic images which can be uneconomical or impossible to collect, and the issue of motion artifacts from stitching images taken at different times.

Innovation Solution

The system synthesizes new panoramic images by projecting nearby images onto a crude 3-D model, allowing for rendering of new views at arbitrary locations and combining them using a blending function, enabling smooth transitions between non-overlapping image sequences and reducing the need for a dense image grid.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If a dense grid of panoramic images is collected to enable realistic navigation through remote locations, then the quality and realism of the virtual tour is improved, but the cost and complexity of data acquisition increases significantly

Engineering Contradiction:

Improverealism of virtual tourVSAvoiddata acquisition complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent creates synthetic copies of panoramic images by projecting and warping existing images to generate new views that did not originally exist. This allows the system to provide realistic navigation through remote locations without physically capturing every viewpoint, thereby reducing data acquisition complexity while maintaining tour realism.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The patent introduces an intermediary computational process that takes existing panoramic images and transforms them into synthetic views through projection and warping operations. This intermediary step enables the system to bridge gaps between captured images and generate seamless virtual tours without requiring dense grid coverage.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Area of stationary object

If panoramic images are stitched together from multiple locations to create continuous views, then the coverage area is improved, but motion artifacts and unrealistic appearances occur due to images being taken at different times

Engineering Contradiction:

Improvecoverage areaVSAvoidrealism of synthesized views

Core Design Contradiction:

Area of stationary objectVSReliability

Solution Approach 1:

The patent applies dynamic warping transformations that adaptively adjust image regions based on their spatial and temporal characteristics. By dynamically adjusting the warping parameters and blending ratios, the system maintains realism in synthesized views while expanding coverage area, effectively handling images taken at different times without prominent motion artifacts.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent applies different warping and blending strategies to different regions of the panoramic image based on local characteristics. Regions with significant motion artifacts receive different treatment compared to stable regions, allowing the system to maintain high realism in static areas while still achieving broad coverage through synthesized views.

Inventive Principle:

Principle #3Local quality

3Productivity

If a crude 3-D model is used for projection instead of a detailed model, then the processing speed and simplicity are improved, but the accuracy of view synthesis may be reduced

Engineering Contradiction:

Improveprocessing speedVSAvoidview synthesis accuracy

Core Design Contradiction:

ProductivityVSManufacturing precision

Solution Approach 1:

The patent applies partial warping transformations only to the necessary regions of the panoramic images rather than transforming entire images. This selective approach maintains processing speed with crude 3-D models while achieving sufficient view synthesis accuracy for the intended application, avoiding unnecessary computational overhead.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS9369679B2System and process for projecting location-referenced panoramic images into a 3-D environment model and rendering panoramic images from arbitrary viewpoints within the 3-D environment model

Publication Date: 2016.06.14 THE BOARD OF TRUSTEES OF THE LELAND STANFORD JUNIOR UNIV

AI summary

A system and process of morphing location-referenced panoramic images into views at nearby locations. When using panoramic images in an interactive tour, a user might desire to see the environment from viewpoints for which no panoramic images are available. This invention makes this possible. It enables a user to experience views from arbitrary locations in the environment, so as long as one or more panoramic images are available at nearby locations. In particular, this invention makes it possible to combine two non-overlapping geo-referenced panoramic video streams into a new video stream which seamlessly transitions between these streams. When used in a client-server architecture, this invention also makes it possible for the server to transmit a sparse sequence of panoramic images, and provide the user with a dense panoramic video stream, by synthesizing the missing panoramic images. Said system and process is also applicable to incomplete panoramic images, photographs, and video.