Multi-view polarization camera-based virtual lighting state correction system and control method thereof

Abstract

Disclosed is a multi-view polarization camera-based virtual lighting state correction system. The multi-view polarization camera-based virtual lighting state correction system comprises: a plurality of camera units including a general camera and a polarization camera that perform multi-view imaging; and a processor that estimates a light source state on the basis of an image set acquired through the general camera and the polarization camera, corrects the acquired image set on the basis of the estimated light source state, trains a neural radiance fields (NeRF) model by inputting the corrected image set into the NeRF model, and reconstructs a 3D image using the trained NeRF model. Accordingly, accurate lighting information may be obtained to set a virtual lighting state close to an actual environment, and the influence of reflected light or highlights may be reduced such that the NeRF model exhibits stable and predictable performance even in an outdoor environment, and the system may be applied to various environments, such as images containing objects with high reflectivity, in addition to outdoor environments, thereby enhancing versatility.

Description

Multi-view Polarized Camera-based Virtual Lighting State Correction System and Control Method

[0001] The present invention relates to a virtual lighting state correction system based on a multi-view polarization camera and a control method thereof, wherein the system and control method estimate the light source state based on an image captured through a polarization camera and correct the image.

[0002] This invention is a study conducted by the Korea Electronics Technology Institute under the research project title "Development of In-Camera Based Interactive Digital VFX Content Production Pipeline Technology" for the research project of the "Culture, Sports and Tourism R&D Support Program," with support from the Ministry of Culture, Sports and Tourism. (Project No. 1375027514, Project Unique Number RS-2023-00216939)

[0003] The emergence of Augmented Reality (AR) and Virtual Reality (VR) has created a demand for high-quality displays of 3D content (e.g., humans, characters, actors, animals, etc.) using performance capture equipment (e.g., cameras and video equipment). Recently, real-time performance capture systems have enabled new use cases for telepresence, augmented video, and live performance broadcasting (in addition to offline multi-view performance capture systems), many of which adaptively provide views of computer-generated scenes based on the viewer's position and perspective.

[0004] In particular, NeRF (Neural Radiance Fields) is a neural network-based computer vision technology for generating and rendering 3D scenes. It is a technology that models 3D scenes of the real world and enables viewing the scene from a new perspective. Recently, Novel View Synthesis technology, which generates scenes from every viewpoint of an object rather than generating a 3D model of the object, has been widely developed.

[0005] However, while NeRF technology generally demonstrates excellent performance in restoring 3D scenes using images taken from multiple viewpoints, it has problems related to lighting changes, particularly in outdoor environments.

[0006] For example, when shooting outdoors, the position and brightness of the sun change depending on the time of day and weather, causing the lighting conditions in each image to vary significantly, which hinders the NeRF from learning consistent 3D information between images. Additionally, in outdoor environments, light reflected from glossy surfaces or objects generates unnecessary highlights or reflections, causing the NeRF to learn distorted actual colors or structures of the images. Furthermore, since these reflections vary depending on the camera angle, they cause inconsistencies between multi-viewpoint data, and there are limitations in effectively correcting complex lighting changes outdoors due to insufficient light source information.

[0007] Accordingly, there is a growing need to enable NeRF systems to be utilized through highly accurate learning even in environments with unfavorable lighting conditions.

[0008] The objective of the present invention is to provide a multi-view polarization camera-based virtual lighting state correction system and a control method thereof that corrects an image by estimating the light source state based on an image captured through a polarization camera.

[0009] A virtual lighting state correction system based on a multi-view polarization camera according to one embodiment of the present invention for achieving such objectives includes a plurality of camera units including a general camera and a polarization camera that perform multi-view shooting, and a processor that estimates a light source state based on an image set obtained through the general camera and the polarization camera, corrects the obtained image set based on the estimated light source state, inputs it into a Neural Radiance Fields (NeRF) model for training, and restores a 3D image through the trained NeRF model.

[0010] Here, the image set may include a first image captured through a general camera and a second image captured through a polarizing camera at each of the multiple viewpoints.

[0011] In addition, the processor can obtain light source direction and intensity information and reflected light information based on an image captured through the polarizing camera, and estimate the state of the light source based on the obtained light source direction and intensity information and reflected light information.

[0012] In addition, the processor can set a virtual lighting state based on the estimated light source state and correct the lighting state of the image set in correspondence with the set virtual lighting state.

[0013] In addition, the processor can correct the lighting conditions of the image set to convert each image included in the image set into an image taken in the same lighting environment.

[0014] In addition, the processor can input the image set with the corrected lighting state into the NeRF (Neural Radiance Fields) model to learn the direction and intensity information of the acquired light source and the reflected light information.

[0015] In addition, the processor can restore a 3D image that reflects the same lighting conditions by adjusting the brightness and color of each image based on the polarized image input through the learned NeRF model.

[0016] Meanwhile, a control method for a virtual lighting state correction system based on a multi-view polarization camera according to one embodiment of the present invention includes the steps of: obtaining a multi-view image set by performing multi-view shooting through a plurality of camera units including a general camera and a polarization camera; estimating a light source state based on the obtained image set; correcting the obtained image set based on the estimated light source state; inputting the corrected image set into a NeRF (Neural Radiance Fields) model for training; and restoring a 3D image through the trained NeRF model.

[0017] Here, the image set may include a first image captured through a general camera and a second image captured through a polarizing camera at each of the multiple viewpoints.

[0018] In addition, the step of estimating the light source state may obtain light source direction and intensity information and reflected light information based on an image captured through the polarizing camera, and estimate the light source state based on the obtained light source direction and intensity information and reflected light information.

[0019] In addition, the step of correcting the acquired image set may set a virtual lighting state based on the estimated light source state and correct the lighting state of the image set in correspondence with the set virtual lighting state.

[0020] Meanwhile, a computer-readable recording medium according to one embodiment of the present invention may include a program for executing a control method for a virtual illumination state correction system based on a multi-view polarization camera on a computer.

[0021] According to various embodiments of the present invention as described above, accurate lighting information can be obtained to set the virtual lighting state to be close to the actual environment, and the influence of reflected light or highlights can be reduced so that the NeRF model can exhibit stable and predictable performance even in outdoor environments. Furthermore, the versatility is enhanced as it can be applied not only to outdoor environments but also to various environments such as images containing objects with large reflected light.

[0022] FIG. 1 is a diagram illustrating the configuration of a virtual lighting state correction system based on a multi-view polarization camera according to one embodiment of the present invention.

[0023] FIG. 2 is a diagram illustrating an image set obtained through a multi-view polarization camera according to an embodiment of the present invention.

[0024] Figures 3 to 5 are diagrams illustrating the learning process of the NeRF model.

[0025] FIG. 6 is a flowchart illustrating a control method for a virtual lighting state correction system based on a multi-view polarization camera according to an embodiment of the present invention.

[0026] Figure 7 is a diagram illustrating the specific configuration of a multi-view polarization camera-based virtual lighting state correction system illustrated in Figure 1.

[0027] FIG. 8 is a drawing relating to a software module stored in a storage unit according to an embodiment of the present invention.

[0028] The present invention will be described in more detail below with reference to the drawings. Furthermore, in describing the present invention, detailed descriptions of related known functions or configurations are omitted if it is determined that such detailed descriptions would unnecessarily obscure the essence of the invention. Additionally, the terms described below are defined considering their functions in the present invention, and these may vary depending on the intentions or relationships of the user or operator. Therefore, their definitions should be based on the content throughout this specification.

[0029] FIG. 1 is a diagram illustrating the configuration of a virtual lighting state correction system based on a multi-view polarization camera according to one embodiment of the present invention.

[0030] Referring to FIG. 1, a virtual lighting state correction system (100) based on a multi-view polarization camera according to one embodiment of the present invention may include a plurality of camera units (110) and a processor (120).

[0031] Here, the multi-view polarization camera-based virtual lighting state correction system (100) may be implemented as an electronic device, server, desktop, portable terminal device, etc., having a separate configuration, or may be implemented as embedded software or a software module.

[0032] Additionally, a processor (120) according to one embodiment of the present invention may be understood as a configuration unit comprising hardware and / or software for performing computing operations. For example, the processor (120) may read a computer program and perform data processing for machine learning. The processor (120) may process computational processes such as processing input data for machine learning, extracting features for machine learning, and calculating errors based on backpropagation. A processor (120) for performing such data processing may include a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA). Since the above-described types of processors (120) are merely examples, the types of processors (120) may be configured in various ways within a range understandable to those skilled in the art based on the contents of this disclosure.

[0033] Additionally, a storage unit (130) according to one embodiment of the present invention may be understood as a configuration unit comprising hardware and / or software for storing and managing data processed in a multi-view polarization camera-based virtual lighting state correction system (100). That is, the storage unit (130) may store data of any form generated or determined by the processor (120) and data of any form received by the network unit.

[0034] For example, the storage unit (130) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory, RAM (random access memory), SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory, magnetic disk, and optical disk. Additionally, the storage unit (130) may include a database system that controls and manages data in a predetermined system. Since the above-described types of the storage unit (130) are merely examples, the types of the storage unit (130) may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

[0035] The storage unit (130) can structure and organize data, combinations of data, and program code executable by the processor (120) that are necessary for the processor (120) to perform calculations. For example, algorithms or programs related to NeRF (Neural Radiance Fields) models may be stored in the storage unit (130). Additionally, the storage unit (130) may store NeRF (Neural Radiance Fields) models, program code that operates to perform learning of the NeRF (Neural Radiance Fields) models, program code that operates to perform inference according to the purpose of use of a multi-view polarization camera-based virtual lighting state correction system (100) that corrects an image set acquired based on the estimated light source state of the NeRF (Neural Radiance Fields) models, and processed data generated as the program code is executed.

[0036] Meanwhile, the plurality of camera units (110) may include a general camera and a polarizing camera that perform multi-view shooting.

[0037] And, the processor (120) estimates the light source state based on the image set obtained through a general camera and a polarizing camera, corrects the image set obtained based on the estimated light source state, inputs it into a Neural Radiance Fields (NeRF) model for training, and can restore the 3D image through the trained NeRF model.

[0038] Here, the image set may include a first image captured through a normal camera and a second image captured through a polarizing camera at each of the multiple viewpoints.

[0039] FIG. 2 is a diagram illustrating an image set obtained through a multi-view polarization camera according to an embodiment of the present invention.

[0040] Referring to FIG. 2, the configuration of a multi-view camera shown on the left is illustrated, and a camera unit (210) according to one embodiment of the present invention includes at least one general camera (211) and a polarizing camera (212), and such a camera unit (210) including at least one general camera (211) and a polarizing camera (212) is configured as a set, and there are multiple such camera units (210, 230).

[0041] And, the image set (210', 230') obtained through the general camera (211) and the polarizing camera (212) may include a first image (221) taken through the general camera (211) and a second image (222) taken through the polarizing camera (212) at each of the multiple viewpoints.

[0042] Additionally, the processor (120) can obtain light source direction and intensity information and reflected light information based on an image captured through a polarizing camera (212), and can estimate the state of the light source based on the obtained light source direction and intensity information and reflected light information.

[0043] Specifically, since the polarization camera (212) can detect the polarization of light, the processor (120) can reduce the influence of reflected light and highlights at specific angles and accurately identify the main light source information.

[0044] For example, the processor (120) can extract information about the main light source by correcting the reflected light and highlight effects at a specific angle according to the polarization of the light detected through the polarization camera (212), and specifically, can extract information about the direction and intensity of the main light source and the reflected light caused by the main light source.

[0045] And, the processor (120) can estimate information such as the position, intensity, direction, and intensity change of the main light source based on the direction and intensity of the extracted main light source and the direction and intensity of the reflected light on the subject.

[0046] And, the processor (120) can set a virtual lighting state based on the estimated light source state and correct the lighting state of the image set in response to the set virtual lighting state.

[0047] Specifically, the processor (120) can correct the lighting conditions of the image set so that each image included in the image set can be converted into an image taken in the same lighting environment.

[0048] Here, the virtual lighting state refers to a unified lighting state extracted based on the first image (221) and the second image (222) obtained through the general camera (211) and the polarizing camera (212), respectively, and may, for example, mean a state in which the direction and intensity of the main light source and the direction and intensity of the reflected light caused by the light source of the set direction and intensity are set to the same lighting conditions.

[0049] And, the processor (120) sets a unified lighting state extracted based on the first image (221) and the second image (222) as a virtual lighting state, and corrects all images of the image set, namely the first image (221) and the second image (222), based on the set virtual lighting state so that all images can be converted to appear as if they were taken in a lighting environment under the same conditions.

[0050] And, the processor (120) can input an image set with corrected lighting conditions into a NeRF (Neural Radiance Fields) model to learn the direction and intensity information of the acquired light source and the reflected light information.

[0051] Specifically, the processor (120) can train a NeRF model by taking an image set with corrected lighting conditions as input and light source direction and intensity information and reflected light information as output, and can restore a 3D image with the same lighting conditions reflected by adjusting the brightness and color of each image based on the input polarized image through the trained NeRF model.

[0052] For example, the processor (120) can adjust the brightness and color of each of the images included in the image set (210', 230') obtained through the general camera (211) and the polarizing camera (212) based on the second image (222), which is a polarizing image input through the learned NeRF model, to be the same lighting condition, thereby restoring a 3D image that reflects a consistent lighting condition even in complex lighting conditions such as an outdoor environment.

[0053] Additionally, the processor (120) can apply a normalization algorithm that reflects the direction and intensity information of the light source and the reflected light information to adjust the brightness and color of each image based on the direction and intensity information of the light source and the reflected light information obtained through the polarization camera, thereby reducing unnecessary reflected light and shadows and preserving the details of the image to normalize the training data of the NeRF model.

[0054] Figures 3 to 5 are diagrams illustrating the learning process of the NeRF model.

[0055] Generally, by inputting images captured from various angles and their camera parameters into a NeRF model, a new view can be output through specific operations.

[0056] The formula illustrated in Fig. 3 illustrates the input and output of the NeRF model, where the 3D position of an object and the direction the object is facing are used as input values, and the RGB color values ​​at the corresponding coordinates and Density, which can be viewed as the inverse of transparency, are output values. Accordingly, the color values ​​and transparency of a specific location at a specific angle can be predicted through the NeRF model.

[0057] In the formula shown in Fig. 3, x, y, and z constituting the input values ​​represent position information in 3D space, θ and Φ represent direction information of the object, RGB constituting the output values ​​represent color information, and σ represents a density value which is the inverse of transparency.

[0058] Meanwhile, referring to Fig. 4, the architecture of the NeRF model is illustrated and consists of an MLP (Multiple Perceptron). The input value is the 3D position (x, y, z) of an object, and the 3D position information of the object is transmitted from the input to the 5th layer. Then, at the 5th layer, the 3D position information of the object is combined once more, which can be thought of as a kind of skip connection. Then, at the 9th layer, the density (σ) is output, and the direction information (d, θ, Φ) for this value is input again, and finally, the RGB value is output.

[0059] This is because density values ​​are related to the object's position and can be predicted based solely on positional information regardless of the viewing angle, whereas RGB values ​​can vary depending on the viewing angle, so directional information can be added in the final layer to predict the RGB values.

[0060] Meanwhile, since the NeRF model uses a supervised learning method, ground truth data is required for training. As image inputs represent objects at various viewpoints, the model can be trained to predict existing views, and training can proceed by calculating the loss between the predicted value and the actual value.

[0061] Referring to FIG. 5, regarding volume rendering of the NeRF model, a ray is emitted from the camera center of a virtual view to be newly created toward an object, and multiple points are sampled along this ray (step (a) of FIG. 5). Then, the RGB color and density values ​​at each coordinate are predicted using the MLP model of FIG. 4, wherein the density value is low at locations where there is no object and high at locations where there is an object.

[0062] The RGB and density values ​​of all points on the sampled ray are projected back onto the desired view, and the appropriate final color value can be calculated through specific operations. In this process, more weight can be assigned to the first object the ray encounters.

[0063] In the training process of the NeRF model described above, an image set with corrected lighting conditions, specifically viewpoint coordinate information for each image, is used as input, and the model can be trained with RGB values ​​and density values ​​related to the direction and intensity of the light source and reflected light information in each image as output. Through the NeRF model trained in this way, the brightness and color of each image can be adjusted.

[0064] FIG. 6 is a flowchart illustrating a control method for a virtual lighting state correction system based on a multi-view polarization camera according to an embodiment of the present invention.

[0065] Referring to FIG. 6, a control method for a virtual lighting state correction system based on a multi-view polarization camera according to an embodiment of the present invention includes the steps of: obtaining a multi-view image set by performing multi-view shooting through a plurality of camera units including a general camera and a polarization camera (S610); estimating a light source state based on the obtained image set (S620); correcting the obtained image set based on the estimated light source state (S630); inputting the corrected image set into a NeRF (Neural Radiance Fields) model for training (S640); and restoring a 3D image through the trained NeRF model (S650).

[0066] Here, the image set may include a first image captured through a normal camera and a second image captured through a polarizing camera at each of the multiple viewpoints.

[0067] Additionally, the step of estimating the light source state (S620) can obtain light source direction and intensity information and reflected light information based on an image captured through a polarizing camera, and estimate the light source state based on the obtained light source direction and intensity information and reflected light information.

[0068] Additionally, the step of correcting the acquired image set (S630) can set a virtual lighting state based on the estimated light source state and correct the lighting state of the image set in correspondence with the set virtual lighting state.

[0069] Meanwhile, as described above, a computer-readable recording medium may be provided that records a program for executing on a computer a control method for a multi-view polarization camera-based virtual lighting state correction system according to one embodiment of the present invention.

[0070] Figure 7 is a diagram illustrating the specific configuration of a multi-view polarization camera-based virtual lighting state correction system illustrated in Figure 1.

[0071] Referring to FIG. 7, a multi-view polarization camera-based virtual lighting state correction system (100) may include a plurality of camera units (110), a processor (120), and a storage unit (130).

[0072] The processor (120) controls the overall operation of the multi-view polarization camera-based virtual lighting state correction system.

[0073] Specifically, the processor (120) includes RAM (121), ROM (122), main CPU (123), graphics processing unit (124), first to n interfaces (125-1 to 125-n), and a bus (126).

[0074] RAM (121), ROM (122), main CPU (123), graphics processing unit (124), first to n interfaces (125-1 to 125-n), etc. can be connected to each other via a bus (126).

[0075] The first to n interfaces (125-1 to 125-n) are connected to the various components described above. One of the interfaces may be a network interface connected to an external device through a network.

[0076] The main CPU (123) accesses the storage unit (130) and performs booting using the O / S stored in the storage unit (130). Then, it performs various operations using various programs, content, data, etc. stored in the storage unit (150).

[0077] In particular, the main CPU (123) estimates the light source state based on an image set obtained through a general camera and a polarizing camera, corrects the image set obtained based on the estimated light source state, inputs it into a Neural Radiance Fields (NeRF) model for training, and can restore a 3D image through the trained NeRF model.

[0078] A set of instructions for booting the system is stored in the ROM (122). When a turn-on command is input and power is supplied, the main CPU (123) copies the O / S stored in the storage unit (150) to the RAM (121) according to the instructions stored in the ROM (122), and executes the O / S to boot the system. When booting is complete, the main CPU (123) copies various application programs stored in the storage unit (150) to the RAM (121), and executes the application programs copied to the RAM (121) to perform various operations.

[0079] The graphics processing unit (124) generates a screen containing various objects such as icons, images, and text using a calculation unit (not shown) and a rendering unit (not shown). The calculation unit (not shown) calculates attribute values ​​such as coordinate values, shape, size, and color for each object to be displayed according to the layout of the screen based on a received control command. The rendering unit (not shown) generates a screen of various layouts containing objects based on the attribute values ​​calculated by the calculation unit (not shown).

[0080] In particular, the graphics processing unit (124) can implement objects generated by the main CPU (123) into a GUI (Graphic User Interface), icon, user interface screen, etc.

[0081] Meanwhile, the operation of the above-described processor (120) can be performed by a program stored in the storage unit (130).

[0082] The storage unit (130) stores various data, such as an O / S (Operating System) software module for operating a multi-view polarization camera-based virtual lighting state correction system (100) and various multimedia content.

[0083] In particular, the storage unit (130) may include a software module for estimating the light source state based on an image set obtained through a general camera and a polarizing camera, correcting the image set obtained based on the estimated light source state, inputting it into a Neural Radiance Fields (NeRF) model for training, and restoring a 3D image through the trained NeRF model.

[0084] FIG. 8 is a drawing relating to a software module stored in a storage unit according to an embodiment of the present invention.

[0085] Referring to FIG. 8, the storage unit (130) may store programs such as a light source state estimation module (131), an image set correction module (132), a NeRF model learning module (133), and a 3D image restoration module (134).

[0086] Meanwhile, the operation of the processor (120) described above can be performed by a program stored in the storage unit (130). Below, the detailed operation of the processor (120) using the program stored in the storage unit (130) will be explained in detail.

[0087] The light source state estimation module (131) can estimate the light source state based on an image set obtained through a general camera and a polarizing camera.

[0088] Additionally, the image set correction module (132) can correct the acquired image set based on the estimated light source state.

[0089] Additionally, the NeRF model learning module (133) can learn the direction and intensity information of the acquired light and the reflected light information when an image set with corrected lighting conditions is input.

[0090] Additionally, the 3D image restoration module (134) can restore a 3D image that reflects the same lighting conditions by adjusting the brightness and color of each image based on the polarized image input through the learned NeRF model.

[0091] Meanwhile, a non-transitory computer-readable medium storing a program that sequentially performs the control method according to the present invention may be provided.

[0092] A non-transient readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, the various applications or programs described above may be stored and provided on non-transient readable media such as CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, and ROMs.

[0093] In addition, although the bus is not shown in the block diagram above illustrating a virtual lighting state correction system based on a multi-view polarization camera, a processor such as a CPU or a microprocessor may be further included to estimate the light source state based on an image set acquired through a general camera and a polarization camera, correct the acquired image set based on the estimated light source state, input it into a Neural Radiance Fields (NeRF) model for training, and restore a 3D image through the trained NeRF model.

[0094] Furthermore, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the invention as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention.

Claims

1. In a virtual lighting state correction system based on a multi-viewpoint polarization camera, A plurality of camera units including a general camera and a polarizing camera that perform multi-view shooting; and A multi-view polarization camera-based virtual lighting state correction system comprising: a processor that estimates a light source state based on an image set acquired through the above-mentioned general camera and polarization camera, corrects the acquired image set based on the estimated light source state, inputs it into a Neural Radiance Fields (NeRF) model for training, and restores a 3D image through the trained NeRF model.

2. In Paragraph 1, The above image set is, A multi-view polarization camera-based lighting state correction system comprising a first image captured through a general camera and a second image captured through a polarization camera at each of the multi-views.

3. In Paragraph 2, The above processor is, A multi-view polarization camera-based lighting state correction system that obtains light source direction and intensity information and reflected light information based on an image captured through the polarization camera, and estimates the light source state based on the obtained light source direction and intensity information and reflected light information.

4. In Paragraph 3, The above processor is, A multi-view polarization camera-based lighting state correction system that sets a virtual lighting state based on the estimated light source state and corrects the lighting state of the image set in correspondence with the set virtual lighting state.

5. In Paragraph 4, The above processor is, A multi-view polarization camera-based lighting condition correction system that corrects the lighting conditions of the above image set and converts each image included in the above image set into an image taken in the same lighting environment.

6. In Paragraph 5, The above processor is, A multi-view polarization camera-based lighting state correction system that inputs an image set with the above lighting state corrected into the NeRF (Neural Radiance Fields) model to learn the direction and intensity information of the acquired light source and reflected light information.

7. In Paragraph 6, The above processor is, A multi-view polarization camera-based lighting state correction system that restores a 3D image reflecting the same lighting state by adjusting the brightness and color of each image based on a polarization image input through the above-mentioned learned NeRF model.

8. A control method for a virtual lighting state correction system based on a multi-viewpoint polarization camera, A step of obtaining a multi-view image set by performing multi-view shooting through a plurality of camera units including a general camera and a polarizing camera; A step of estimating the light source state based on the above-mentioned acquired image set; A step of correcting the acquired image set based on the estimated light source state; The step of inputting the above-mentioned corrected image set into a NeRF (Neural Radiance Fields) model for training; and A control method for a multi-view polarization camera-based virtual lighting state correction system comprising the step of restoring a 3D image through the above-mentioned learned NeRF model.

9. In Paragraph 8, The above image set is, A control method for a multi-view polarization camera-based virtual lighting state correction system comprising a first image captured through a general camera and a second image captured through a polarization camera at each of the multi-views.

10. In Paragraph 9, The step of estimating the light source state above is, A control method for a multi-view polarization camera-based virtual lighting state correction system, wherein the direction and intensity information of a light source and reflected light information are obtained based on an image captured through the polarization camera, and the state of the light source is estimated based on the obtained direction and intensity information of the light source and reflected light information.

11. In Paragraph 10, The step of correcting the above-mentioned acquired image set is, A control method for a multi-view polarization camera-based virtual lighting state correction system, wherein a virtual lighting state is set based on the estimated light source state and a lighting state of an image set is corrected in correspondence with the set virtual lighting state.

12. A computer-readable recording medium having a program stored on it for executing on a computer a control method for a multi-view polarization camera-based virtual lighting state correction system described in any one of paragraphs 8 through 11.

Images