Virtual Viewpoint Image Generation Using Predicted Subject Position
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for generating virtual viewpoint images struggle when there are multiple subjects in the imaging area, leading to delays and potential failure in generating the images.
Innovation Solution
An image processing apparatus that predicts the virtual viewpoint and the position of a three-dimensional subject model in subsequent frames, determines the appropriate image capturing apparatus based on these predictions and image capturing parameters, and generates the virtual viewpoint image using the captured images from the determined apparatus.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If captured images are sequentially switched and read out from the database according to virtual viewpoint movement, then the virtual viewpoint image can be generated dynamically, but it takes a long time to generate the virtual viewpoint image causing delay
Solution Approach 1:
The system performs preliminary actions by predicting the virtual viewpoint position for the next frame in advance and pre-determining which captured image should be used. This allows the image generation process to start before the actual viewpoint change occurs, significantly reducing the delay in generating dynamic virtual viewpoint images.
2Productivity
If images to be used for generating the virtual viewpoint image are determined based only on prediction of the virtual viewpoint, then the time for generating the virtual viewpoint image can be reduced, but the virtual viewpoint image cannot be generated in some cases when there are multiple subjects
Solution Approach 1:
The system uses feedback by detecting the actual position of the three-dimensional subject model and comparing it with the predicted virtual viewpoint. Based on this feedback, the system adjusts the selection of captured images to ensure that subjects are properly included in the generated virtual viewpoint image, thereby maintaining both speed and reliability.
Solution Approach 2:
The system dynamically adapts the image selection process by continuously updating the predicted subject position and adjusting which captured images are used. This dynamic adjustment ensures that the system can handle multiple subjects effectively while maintaining high generation speed through efficient prediction algorithms.
3Measurement precision
If the image capturing apparatuses are sequentially changed based on virtual viewpoint movement, then the appropriate apparatus can be selected for each viewpoint, but it causes delay in generating the virtual viewpoint image
Solution Approach 1:
The system determines in advance which image capturing apparatus should be used for the next virtual viewpoint based on prediction. This preliminary determination eliminates the need for sequential apparatus switching during actual image generation, thereby maintaining high apparatus selection accuracy while significantly improving generation speed.
Data Source
AI summary
An image processing apparatus predicts a virtual viewpoint in a second frame subsequent to a first frame in a virtual viewpoint image and a position of a three-dimensional subject model in the second frame. The image processing apparatus determines an image capturing apparatus from which a captured image to be used for generating the second frame is obtained from among the plurality of image capturing apparatuses based on the predicted virtual viewpoint, the predicted position of the three-dimensional subject model, and image capturing parameters of the image capturing apparatuses, and generates the virtual viewpoint image based on the captured image for the second frame obtained from the determined image capturing apparatus, the three-dimensional subject model corresponding to the second frame, and the virtual viewpoint information corresponding to the second frame.