Virtual Viewpoint Generation via Occlusion Removal
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional video broadcasts of live events are limited by camera positions, often occluding important action and failing to capture the most critical viewpoints due to safety and practical constraints.
Innovation Solution
A system and method for providing images of an event from different virtual viewpoints by automatically detecting and removing occluding objects, such as players, and using blob detection techniques to identify and maintain relevant objects, allowing for the creation of textured 3D models and virtual viewpoints that differ from the camera's perspective.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If cameras are positioned in locations that capture action from desired viewpoints, then viewing quality is improved, but safety and practical constraints are violated
Solution Approach 1:
The patent creates a virtual copy of the camera viewpoint by processing images from existing cameras. The system detects lines and objects in images, removes occluding objects, and generates a synthesized virtual viewpoint that replicates what would be seen from an inaccessible camera position, thus achieving improved viewing quality without violating safety constraints
Solution Approach 2:
The patent replaces the mechanical camera positioning system with an image processing system. Instead of physically moving cameras to desired viewpoints (which would violate safety constraints), the system uses computational methods to create virtual viewpoints by detecting and removing occlusions from existing camera images
2Measurement precision
If occluding objects are removed from images, then line visibility is improved, but image processing complexity increases
Solution Approach 1:
The patent segments the image processing into distinct functional steps: detecting lines in the image, identifying occluding objects, removing objects, and repairing lines. This segmentation allows each step to be optimized independently, managing overall complexity while improving line visibility through systematic processing
Solution Approach 2:
The patent performs preliminary line detection before object removal. By first identifying and marking lines in the image, the system can then selectively remove occluding objects while preserving line information, and finally repair the lines in the modified regions. This preliminary action ensures line visibility is maintained throughout the complex processing pipeline
Data Source
AI summary
In one aspect, lines in image data of an event are automatically found and repaired. For example, the event may be a sporting event which is played on a field, and the line segment is a field line on the field which may be obscured by a player, game ball or other object. The line segment is automatically detected in a mask image, and a portion of the line segment which is occluded by the object is automatically determined, and the object is automatically removed. The line segment can also be repaired. Optionally, a virtual viewpoint of the event is provided from the image, with the line repaired and the object removed. In another aspect, an object in an image of an event is automatically located by detecting blobs in the image which meet at least one specified criterion, such as size, aspect ratio, density or color profile.