Virtual Viewpoint Image Generation for Underwater Sports

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional methods for generating virtual viewpoint images, especially in underwater sports, fail to accurately represent the virtual viewpoint position due to the lack of three-dimensional modeling of water surfaces, leading to confusion about the viewpoint's location relative to the water surface.

Innovation Solution

An image processing apparatus that acquires viewpoint information and generates virtual viewpoint images by determining the positional relationship between the virtual viewpoint and the water surface, adding images representing the water surface to the virtual viewpoint image to clearly indicate the viewpoint's position, either above or below the water.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If background subtraction information is used to generate virtual viewpoint images in underwater sports, then the processing speed and simplicity are improved, but the accuracy of viewpoint position recognition deteriorates because water surfaces cannot be properly modeled

Engineering Contradiction:

Improveimage generation speedVSAvoidviewpoint position recognition accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The space is segmented into multiple regions (water region, air region, pool region) with different boundary characteristics. Each region is processed differently: water surfaces use refraction-based boundary generation while air surfaces use reflection-based boundary generation, allowing accurate viewpoint positioning without compromising processing efficiency

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The boundary generation process changes parameters based on the medium type. For water boundaries, the refractive index parameter is applied; for air boundaries, the reflection parameter is applied. This allows the system to maintain high processing speed while achieving accurate viewpoint position recognition by adapting to different physical conditions

Inventive Principle:

Principle #35Parameter changes

2Measurement precision

If three-dimensional models of water surfaces are generated using laser measurement, then the viewpoint position recognition is improved, but the manufacturing complexity and cost increase significantly

Engineering Contradiction:

Improvewater surface positioning accuracyVSAvoidthree-dimensional modeling complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

Instead of creating complex three-dimensional models of water surfaces using laser measurement, the invention generates virtual boundary images that copy the essential visual characteristics of water surfaces. These boundary images are synthesized from captured images using refraction calculations, providing sufficient positioning information without the complexity of full 3D modeling

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The invention replaces the mechanical laser measurement system with an optical-computational approach. By using captured images and applying refraction-based boundary generation algorithms, the system achieves water surface positioning without requiring complex physical measurement equipment

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Ease of operation

If virtual viewpoint images are generated without representing water surfaces, then the processing simplicity is maintained, but the realism and user recognition of viewpoint position deteriorate

Engineering Contradiction:

Improveprocessing simplicityVSAvoidviewpoint position recognition reliability

Core Design Contradiction:

Ease of operationVSReliability

Solution Approach 1:

The invention applies partial action by generating only the essential boundary images of water surfaces rather than complete three-dimensional models. This partial representation provides sufficient information for viewpoint position recognition while maintaining processing simplicity and avoiding unnecessary complexity

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS11277596B2Image processing apparatus, image processing method, and storage medium

Publication Date: 2022.03.15 CANON KK

AI summary

An image processing apparatus for generating a virtual viewpoint image based on a plurality of images obtained by capturing an object in an image-capturing region from a plurality of directions by using a plurality of image capturing apparatuses includes a viewpoint acquisition unit configured to acquire viewpoint information indicating a position of a virtual viewpoint, and a generation unit configured to generate the virtual viewpoint image corresponding to the position of the virtual viewpoint indicated by the acquired viewpoint information, the virtual viewpoint image including at least one of an image depending on a positional relationship between a boundary and the virtual viewpoint and/or an image representing the boundary, the boundary being between an inside and an outside of a predetermined region included in the image-capturing region.