Rendering method and device including light source recovery

Abstract

An electronic device according to the present disclosure may: acquire a plurality of two-dimensional images captured from a plurality of viewpoints for a first scene including a first light source by using a camera; generate a color set including some of a plurality of colors included in the plurality of two-dimensional images; predict the structure and radiance of a 3D space for the first scene through volume rendering for the plurality of two-dimensional images; select a first color from the color set as a light source candidate color; predict a weight of the color set and an object-specific color at each pixel in the 3D space through volume rendering; predict a scene component of the 3D space through physically based rendering for the plurality of two-dimensional images; identify that the first color is a color of the first light source when an error between pixel values of the first scene and the plurality of two-dimensional images is less than a first threshold value as the result of the physically based rendering; re-select a light source candidate color from the color set when the error between the pixel values of the first scene and the plurality of two-dimensional images is greater than the first threshold value as the result of the physically based rendering; and generate an image of a first viewpoint for the 3D space when the physically based rendering is completed.

Description

Rendering method and device including light source restoration

[0001] The present disclosure relates to a rendering method and apparatus including light source restoration.

[0002] Rendering is the process of converting 2D images or animations into 3D models or scenes. This process plays a crucial role in computer graphics and is used in both real-time applications (e.g., games) and non-real-time applications such as movies and animations. Rendering primarily involves methods to realistically represent a scene by calculating various elements, such as lighting sources, camera angles, and material properties. Rendering techniques include, for example, rasterization, ray tracing, and path tracing. Each method can be selected based on a balance between speed and quality. For instance, while ray tracing enables realistic lighting effects, its high computational cost limited its use in real-time environments; however, recent hardware advancements have led to its increasing utilization in games.

[0003] In addition, neural network-based rendering technology has recently been receiving significant attention. This technology focuses on reproducing scenes more efficiently and realistically by utilizing deep learning. A representative example is Neural Radiance Fields (NeRF). NeRF reconstructs 3D scenes using only a few photos and allows for free movement of the viewpoint. NeRF provides high-quality results with less data and computation than conventional rendering methods and is being applied in various fields such as film, virtual reality (VR), and digital twins.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0005] An electronic device according to one embodiment of the present disclosure comprises: a camera; a display; a memory comprising at least one storage medium for storing instructions; and at least one processor comprising a processing circuit; When the above instructions are executed individually or collectively by the at least one processor, the electronic device: acquires a plurality of two-dimensional images captured at multiple viewpoints of a first scene including a first light source using the camera; generates a color set including some of the plurality of colors included in the plurality of two-dimensional images; predicts the structure and radiance of the 3D space for the first scene through volume rendering of the plurality of two-dimensional images; selects a first color among the color set as a candidate color for the light source; predicts the weight of the color set and the object's intrinsic color at each pixel in the 3D space through volume rendering; predicts the scene components of the 3D space through physically based rendering of the plurality of two-dimensional images; and if the error between the pixel values ​​of the first scene and the plurality of two-dimensional images resulting from the physically based rendering is within a first threshold value, identifies that the first color is the color of the first light source, and When the error between the pixel values ​​of the first scene and the plurality of two-dimensional images resulting from the physically based rendering is greater than the first threshold, a light source candidate color is selected again from the set of colors, and when the physically based rendering is completed, it may cause the generation of an image of the first viewpoint in the 3D space.

[0006] A method of an electronic device according to another embodiment of the present disclosure comprises: acquiring a plurality of two-dimensional images captured at a plurality of viewpoints for a first scene including a first light source; generating a color set including some of a plurality of colors included in the plurality of two-dimensional images; predicting the structure and radiance of a 3D space for the first scene through volume rendering of the plurality of two-dimensional images; selecting a first color among the color set as a candidate color for the light source; predicting the weight of the color set and the object's intrinsic color at each pixel in the 3D space through volume rendering; predicting scene components of the 3D space through physically based rendering of the plurality of two-dimensional images; and identifying that the first color is the color of the first light source when the error between the pixel values ​​of the first scene and the plurality of two-dimensional images resulting from the physically based rendering is within a first threshold value. The method may include: returning to the operation of selecting a light source candidate color from the set of colors when the error between the pixel values ​​of the first scene and the plurality of two-dimensional images in the physical-based rendering result is greater than the first threshold value; and generating an image of a first viewpoint in the 3D space when the physical-based rendering is completed.

[0007] A non-transitory computer-readable storage medium storing instructions according to another embodiment of the present disclosure, wherein the instructions, when executed by one or more processors, cause the one or more processors to: acquire a plurality of two-dimensional images taken at a plurality of viewpoints for a first scene including a first light source; generate a color set including some of a plurality of colors included in the plurality of two-dimensional images; predict the structure and radiance of a 3D space for the first scene through volume rendering of the plurality of two-dimensional images; select a first color among the color set as a candidate light source color; predict the weight of the color set and the object's intrinsic color at each pixel in the 3D space through volume rendering; and predict a scene component of the 3D space through physically based rendering of the plurality of two-dimensional images. When the error between the pixel values ​​of the first scene and the plurality of two-dimensional images resulting from the physical-based rendering is within a first threshold value, the operation of identifying that the first color is the color of the first light source; when the error between the pixel values ​​of the first scene and the plurality of two-dimensional images resulting from the physical-based rendering is greater than the first threshold value, the operation of returning to selecting a candidate light source color from the set of colors; and when the physical-based rendering is completed, the operation of generating an image of a first viewpoint in the 3D space may be performed.

[0008] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0009] FIG. 2 is an example of 3D rendering for a single light source multi-view input image of an electronic device according to one embodiment.

[0010] FIG. 3 is a flowchart illustrating a rendering operation including the restoration of a light source of an electronic device according to one embodiment.

[0011] FIG. 4 is a flowchart illustrating the operation of an electronic device generating a set of colors according to one embodiment of the present disclosure.

[0012] FIG. 5 is a flowchart illustrating the operation of an electronic device restoring a light source using a set of colors according to one embodiment of the present disclosure.

[0013] FIGS. 6a and 6b are flowcharts illustrating detailed operations of an electronic device restoring a light source according to one embodiment of the present disclosure.

[0014] FIG. 7 is a flowchart illustrating the operation of an electronic device modifying a light source of a rendering screen according to user input, according to one embodiment of the present disclosure.

[0015] FIGS. 8a and 8b are examples of lighting editing screens according to one embodiment of the present disclosure.

[0016] In the following description, the attached drawings are referenced, and specific examples of implementation are illustrated within the drawings. Additionally, other examples may be used and structural modifications may be made without departing from the scope of the various examples.

[0017] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit the technical features of this disclosure. For example, a component expressed in the singular form should be understood as a concept including singular or plural components unless the context clearly indicates only the singular form.

[0018] In the present disclosure, each of the phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B, or C” may include any one of the items listed with the corresponding phrase, or all possible combinations thereof. The term “and / or” as used in the present disclosure should be understood to encompass any possible combination by one or more of the plurality of items listed with the corresponding term. Terms such as “first,” “second,” “first,” or “second” as used in the present disclosure may be used merely to distinguish a component from another component and do not limit the components in any other aspect (e.g., importance or order).

[0019] Where it is stated that any (e.g., 1st) component is “coupled,” “connected,” “linked,” “coupled,” “supported,” “connected,” or “contacted” with or without the terms “functionally” or “communicationly,” it includes not only cases where the component is directly coupled, connected, linked, coupled, supported, or contacted with the other component, but also cases where it is indirectly coupled, connected, linked, coupled, supported, or contacted through a third component.

[0020] Terms such as "include" or "have" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in this disclosure, and do not preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof. When a component is described as being located "on" another component, this includes not only cases where the component is in contact with the other component, but also cases where another component exists between the two components.

[0021] The expression “configured to” as used in this disclosure may be appropriately substituted, depending on the context, for example, “suitable for,” “capable of,” “designed to,” “modified to,” “made to,” or “capable of.” The term “configured to” does not necessarily mean only that which is “specially designed” in hardware. Instead, in some situations, the expression “device configured to” may mean that the device is “capable of” together with other devices or components. For example, the phrase “device configured (or set) to perform A, B, and C” may mean a device dedicated to performing the said operation, or a general-purpose device capable of performing various operations including said operation.

[0022] Terms used in this disclosure, such as "upper side," "lower side," and "front-rear direction," are defined based on the drawings, and the shape and position of each component are not limited by these terms.

[0023] Although the description in this disclosure is centered on specific embodiments, this disclosure is not limited to such specific embodiments and should be understood to encompass all various modifications, equivalents, and / or substitutions of the various embodiments described in this disclosure. In connection with the description of the drawings, similar reference numerals may be used for similar or related components.

[0024] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0025] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0026] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0027] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0028] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, software (e.g., program (140)) and input data or output data for related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0029] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0030] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0031] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0032] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0033] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0034] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0035] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multi-media interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0036] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0037] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0038] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0039] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0040] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0041] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0042] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or largescale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0043] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0044] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0045] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0046] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0047] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0048] FIG. 2 is an example of 3D rendering for a single light source multi-view input image of an electronic device according to one embodiment.

[0049] An electronic device (101) according to one embodiment can reconstruct a three-dimensional space (220), including restoring a light source, using two-dimensional images (210) of a first scene, and render it to generate an image at a specific point in time.

[0050] An electronic device (101) may receive multi-view 2D images (210). For example, the electronic device (101) may acquire 2D images captured from multiple viewpoints through a camera (e.g., camera module (180) of FIG. 1). Alternatively, the electronic device (101) may receive multi-view 2D images of a specific space or a specific object by user input. In one embodiment, the 2D images (210) may include a single light source. The multi-view images may be LDR (low dynamic range) images acquired through a standard camera. LDR images have a more limited range of colors that can be represented by 8-bit pixel values ​​than HDR (high dynamic range) images. HDR images can represent more colors by 32-bit pixel values, but they can only be acquired through a high-performance camera.

[0051] According to one embodiment, an electronic device (101) can render a 3D space in which a light source has been restored using multi-view general images of a scene containing a light source. The electronic device (101) can create a color set composed of representative colors by selecting some of the multiple colors included in the multi-view images as representative colors. The electronic device (101) can then use the colors included in the color set to represent other colors of the input images. For this reason, the color set can function as a palette.

[0052] An electronic device (101) according to one embodiment can represent and render a 3D scene (220) using artificial intelligence (e.g., deep learning-based Neural Radiance Fields (NeRF)). According to one embodiment, NeRF can represent the result of the accumulation of color and density in a certain space using at least one image captured by at least one camera, and estimate the appearance of the space based on this information. NeRF may include, for example, a model capable of predicting radiance, light intensity, or color at a specific point within the scene included in at least one image.

[0053] An electronic device (101) according to one embodiment can restore a light source in a scene using NeRF, thereby enabling detailed changes to the light source and allowing for precise expression of reflection and shadow effects. Restoration may refer to the process of modeling a scene by performing NeRF on a given multi-view image and training a deep learning model to match the original image. By training to minimize the difference (loss) between the image value and the rendered value, the modeled scene can be made identical to the given image.

[0054] An electronic device (101) according to one embodiment can find the color of a light source within an image using a set of colors. The electronic device (101) places an arbitrary color within the set of colors as a light source candidate, first predicts the shape and color of the 3D space, and, based on the predicted result, predicts the components of a specific scene. If the difference from the input image falls within a defined error range, the light source candidate can be identified as the color of the light source. Since only some of the colors included in the input images are placed as light source candidates, computational complexity can be reduced. The electronic device (101) can separate the influence of the light source by setting different weights for the light source and non-light source colors in the set of colors. The electronic device (101) can distinguish whether the light entering through the camera is reflected by the light source or reflected from the surrounding environment. Since the electronic device (101) has separated the influence of the light source, it can provide the user with a function to change the light source (e.g., color type, intensity, size, shape, location). The user can edit the lighting while viewing a rendering screen of the 3D space, for example, by taking a photo of a specific scene or object and inputting it. The electronic device (101) can distinguish the influence of the light source and re-render based on the user input for editing the lighting.

[0055] FIG. 3 is a flowchart illustrating a rendering operation including the restoration of a light source of an electronic device according to one embodiment.

[0056] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1) can learn multi-view 2D images to restore a light source and render a 3D scene. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0057] In operation 310, an electronic device (101) according to one embodiment may acquire a plurality of 2D images of a first scene. The first scene may include a first light source. The plurality of 2D images may be captured at a plurality of viewpoints. For example, the electronic device (101) may receive 2D images captured at multiple viewpoints under a single light source (the first light source) as input data.

[0058] In operation 320, an electronic device (101) according to one embodiment may generate a color set that includes some of the multiple colors included in multiple two-dimensional images. The color set represents a representative color among the RGB values ​​of each pixel of the input images. That is, the electronic device (101) may represent the color of all pixels in the image using the colors included in the color set. For example, the electronic device (101) may define a weight for each color set and represent each pixel's RGB value using the color included in the color set and the weight. Additionally, the color set may include the color of a light source. The light source may be included in the representative color because its light intensity is stronger than that of other objects.

[0059] In operation 330, an electronic device (101) according to one embodiment can predict the geometry and radiance of a 3D space for a first scene through volume rendering of a plurality of 2D images. Radiance can be expressed as light intensity and color. Through volume rendering, the electronic device (101) can identify the shape and color of a 3D space for a first scene.

[0060] In operation 340, an electronic device (101) according to one embodiment may select a light source candidate color from a set of colors. The electronic device (101) may select any first color among the colors included in the set of colors as a light source candidate color. The electronic device (101) may select a light source candidate color from the remainder, excluding colors that have already been identified as not being light sources. If all colors in the set of colors are identified as not being light sources, the electronic device (101) may determine that there are no light sources in the input images and may not perform light source restoration. The electronic device (101) may provide information to the user that there are no light sources through a display.

[0061] In operation 350, an electronic device (101) according to one embodiment can predict the weight of a color set and the object's unique color at each pixel in 3D space through volume rendering. While volume rendering in operation 330 predicted the structure and color of 3D space, volume rendering in operation 350 can predict each weight of a color set and the object's unique color.

[0062] An electronic device (101) according to one embodiment can predict that the color weight of the light source will gradually increase due to light reflection around the color of the light source. For example, the weight of yellow may change smoothly. The electronic device (101) can predict that the weight of colors other than the color of the light source will be expressed as discrete values. Colors other than the color of the light source represent the color of an object, and since they are not the color of the light source, they may not change smoothly and may have discrete values. For example, red or green may have a weight of 0 near a yellow light source. Reflecting this, the electronic device (101) can distinguish between the color designated as the light source candidate color and the remaining colors in the color set and update the weights differently. While performing volume rendering, the electronic device (101) can set the color designated as the light source candidate color to have a continuous weight value, and the remaining colors not designated as the light source candidate color to have a discrete weight value. The electronic device (101) can separate the influence of the light source through the weights of the color set. That is, the electronic device (101) can distinguish whether the light included in the image, that is, the light entering the camera, is light from a light source or light from the surrounding environment.

[0063] An electronic device (101) according to one embodiment may take into account in the learning process that the color of the light source can be created by combining different colors of a set of colors. For example, if the color of the light source is yellow (R=255, G=255, B=0), the color of the light source (Red (R=255, G=0, B=0) + Green (R=0, G=255, B=0) = Yellow (R=255, G=255, B=0)) can be created by combining red (R=255, G=0, B=0) and green (R=0, G=255, B=0). The electronic device (101) can prevent a given situation where the combination of red and green is given as the color of the light source.

[0064] An electronic device (101) according to one embodiment can learn a unique color at each pixel within each three-dimensional space. By learning the unique color separately, the electronic device (101) can reflect the degree to which light received from a light source is reflected as the object's unique color and visible to the camera. The electronic device (101) can accumulate colors other than the light source's color within a color set by multiplying only the color of the color set and the weight. Since the reflected color may change depending on the color of the object, the electronic device (101) can accumulate the light source's color by multiplying the color of the color set, the weight, and the object's unique color together. For example, if yellow light is shone on a red object, it will appear red. That is, the object's unique color can modify the reflected light. To reflect that the light source's color changes due to the object's color, the electronic device (101) can learn the object's unique color together. Even if light is received from a light source, the reflected color may differ and may not be expressed by the weight of the color set. The influence of the light source may be included in the color set colors other than the light source color, making it impossible to create discrete values. If it is impossible to make the weights of colors that are not of the light source discrete in the color set by learning the unique color together, the electronic device (101) determines that the light from the light source has been changed to another color in the color set due to the object's unique color, and can adjust the learning so that it can be corrected back to the color of the light source.

[0065] In 360 motion, an electronic device (101) according to one embodiment can predict scene components in 3D space through physical-based rendering of a plurality of 2D images. Scene components may include emission, material (albedo, roughness), and light from the surrounding environment (environment map) in 3D space. Based on the given scene components, the electronic device (101) can determine how light will be reflected into the camera by performing physical-based rendering. The light received at a 3D location is determined by the emission level and light from the surrounding environment, and the electronic device (101) can calculate how much of the light received at a 3D location is reflected into the camera using the material of the object. Since the light entering the camera in 350 motion is separated into light from the light source and light from the surrounding environment, the electronic device (101) can calculate the reflection from the light source and the surrounding environment separately in 360 motion. The electronic device (101) can learn so that the reflected light appears identical to the learning images, i.e., multiple two-dimensional images. Since the influence of the light source is separated in the operation, the electronic device (101) learns that the reflection calculated from the light source is identical to the influence of the light source, and the reflection calculated from the surrounding environment is learned so that it is identical to the influence of the surrounding environment.

[0066] In operation 370, an electronic device (101) according to one embodiment can determine whether the error between the value of the physical-based rendering result and the two-dimensional image is within a predetermined first threshold. If the error is within the first threshold, the electronic device (101) can identify that the color designated as the current light source candidate color is the color of the light source.

[0067] If the error is greater than the first threshold, the electronic device (101) identifies that the color currently designated as the light source candidate color is not a light source and returns to operation 340 to select the light source candidate color again from the set of colors.

[0068] In operation 380, when physical-based rendering is completed, the electronic device (101) according to one embodiment can generate an image of a first viewpoint in 3D space and output it through a display. The electronic device (101) can apply an emphasis effect to the restored light source. The emphasis effect can, for example, make the border thicker or apply a glowing effect. The electronic device (101) receives user input to change the viewpoint in 3D space and can generate an image of the changed viewpoint and output it through a display. For example, when a touch or drag action is input to the image of the first viewpoint, the electronic device (101) can output a rendering screen with the viewpoint in 3D space changed.

[0069] An electronic device according to one embodiment of the present disclosure comprises: a camera; a display; a memory; and at least one processor including a processing circuit; When the above memory is executed individually or collectively by the at least one processor, the electronic device enables: to acquire a plurality of two-dimensional images captured at a plurality of viewpoints for a first scene including a first light source using the camera; to generate a color set including some of the plurality of colors included in the plurality of two-dimensional images; to predict the structure and radiance of the 3D space for the first scene through volume rendering of the plurality of two-dimensional images; to select a first color among the color set as a candidate color for the light source; to predict the weight of the color set and the object's intrinsic color at each pixel in the 3D space through volume rendering; to predict the scene components of the 3D space through physically based rendering of the plurality of two-dimensional images; and if the error between the pixel values ​​of the first scene and the plurality of two-dimensional images resulting from the physically based rendering is within a first threshold value, to identify that the first color is the color of the first light source, and the physical When the error between the pixel values ​​of the first scene and the plurality of two-dimensional images resulting from the base rendering is greater than the first threshold, a light source candidate color is selected again from the set of colors, and instructions can be stored to cause the generation of an image of the first viewpoint in the 3D space when the physical base rendering is completed.

[0070] According to one embodiment, the memory can store instructions that, when executed individually or collectively by the at least one processor, cause the electronic device to: include colors corresponding to each vertex of a three-dimensional polygon (convex hull) formed by connecting the outermost boundary points among the points corresponding to a plurality of colors included in the plurality of two-dimensional images in a color space composed of RGB values.

[0071] According to one embodiment, the memory may store instructions that, when executed individually or collectively by the at least one processor, cause the electronic device to: initialize and update the weights for colors other than the color designated as the light source candidate color among the plurality of colors included in the color set to discrete values, and initialize and update the weights for the color designated as the light source candidate color to continuous values.

[0072] According to one embodiment, the memory may store instructions that, when executed individually or collectively by the at least one processor, cause the electronic device to: update the weight for a color other than the color designated as the light source candidate color by selecting the closest value among the determined discrete values ​​while predicting the weight of the color set through the volume rendering.

[0073] According to one embodiment, the memory may store instructions that, when executed individually or collectively by the at least one processor, cause the electronic device to: repeat volume rendering for predicting the weights of the color set and the object's inherent color until the difference between the result of the volume rendering and the plurality of two-dimensional images reaches within a predetermined error.

[0074] According to one embodiment, the memory can store instructions that, when executed individually or collectively by the at least one processor, cause the electronic device to: while predicting the object's inherent color through volume rendering, cause the color designated as the light source candidate color to reflect the color, weight, and inherent color of the color set of colors.

[0075] According to one embodiment, when the memory is executed individually or collectively by the at least one processor, the electronic device may: the scene components of the 3D space may include the luminescence of an object, a material, and light of the surrounding environment.

[0076] According to one embodiment, the memory can store instructions that, when executed individually or collectively by the at least one processor, cause the electronic device to: predict scene components of the 3D space using a surface property prediction model and an albedo prediction model of the 3D space for the physically based rendering.

[0077] According to one embodiment, when the memory is executed individually or collectively by the at least one processor, the electronic device may: display an emphasis effect on the first light source of the image at the first time point and output it through the display.

[0078] According to one embodiment, the memory may store instructions that, when executed individually or collectively by the at least one processor, cause the electronic device to: receive touch or drag user input on a screen that outputs an image at the first time point, and, based on the touch or drag user input, output a rendering screen at a changed time point.

[0079] According to one embodiment, the memory may store instructions that, when executed individually or collectively by the at least one processor, cause the electronic device to: receive user input regarding the first light source included in the image at the first time point output through the display, reflect the light source attributes changed by the user input through physical-based rendering, output the image resulting from the physical-based rendering through the display, and the user input to change at least one of the color, intensity, position, size, or shape of the light source.

[0080] FIG. 4 is a flowchart illustrating the operation of an electronic device generating a set of colors according to one embodiment of the present disclosure.

[0081] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1) can generate a set of colors that serve as a palette based on color information included in an input image. The electronic device (101) can represent other colors included in the input image using colors included in the set of colors. Operations 321 to 324 of FIG. 4 may be included in operation 320 of FIG. 3.

[0082] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0083] 321 In operation, an electronic device (101) according to one embodiment can extract a plurality of colors included in input images. The input images are taken with a standard camera and, for example, may be a plurality of LDR 2D images. The input images may be multi-view images of a specific scene or object including a single light source.

[0084] 322 In operation, an electronic device (101) according to one embodiment can map a plurality of extracted colors to a color space composed of RGB values. The electronic device (101) can, for example, obtain an RGB 3D map to which all pixel RGB values ​​of input images are mapped.

[0085] In operation 323, an electronic device (101) according to one embodiment may select the largest 3D convex hull in an RGB 3D map. The electronic device (101) may obtain the largest convex hull that includes each pixel point of the RGB 3D map. The convex hull may include all pixel RGB values ​​of the image. That is, the convex hull represents the largest convex hull that completely encloses the color distribution of the image in the color space.

[0086] In operation 324, an electronic device (101) according to one embodiment can generate a set of colors at each vertex of a 3D polygon. The set of colors may be referred to as a palette. This is because all pixel RGB values ​​of an input image can be expressed using the colors included in the palette. At this time, since the light source has an intensity strong enough to exert influence within the scene, it can be specifically expressed in an HDR (high dynamic range) image with 32-bit pixel values. However, unlike HDR images, in LDR (low dynamic range) images (e.g., input images) with 8-bit pixel values, the range that can be expressed (0 to 255) is limited, so the color of the light source can be expressed as a boundary value (e.g., 255). For example, if the light source is yellow and has a strong intensity, it can be expressed as (R=255, G=255, B=0), which is the maximum intensity that yellow can have. In LDR images, light sources are represented by their maximum intensity, so the vertices of the convex polygons in the RGB 3D map may contain the color of the light source. However, even in the case of HDR images, if a bright light source outside the range that the HDR image can represent is captured, the convex polygon can be constructed using multi-view images acquired with a single light source turned on. Alternatively, even within an HDR image, since the light source color has a stronger intensity compared to other colors, it may be the maximum value among the colors, even if it is not a boundary value that the HDR image can represent; therefore, there is a high possibility that it corresponds to the vertices when calculating the convex polygon. When the input image is an HDR image versus an LDR image, there may be detailed changes in the light source reconstruction algorithm due to differences in image format.

[0087] The electronic device (101) can represent all pixel values ​​of an RGB 3D map using colors included in a color set (i.e., boundary values ​​corresponding to vertices of an RGB 3D map). Furthermore, the electronic device (101) can not only simply restore colors in a three-dimensional space using the color set, but also identify the color of a light source within the color set and separate the influence of the light source based on the characteristic that the color of the light source is necessarily included in the color set.

[0088] FIG. 5 is a flowchart illustrating the operation of an electronic device restoring a light source using a set of colors according to one embodiment of the present disclosure.

[0089] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1) can verify whether the assumed light source color is an actual light source by assuming that there is a light source within a set of colors and predicting a unique color based on an arbitrarily selected light source color through a learning process. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0090] According to one embodiment, the electronic device (101) may acquire an image taken at multiple points in time with the light turned on (e.g., a multi-point image under a single light source condition) as an input image. For example, the electronic device (101) may create a 3D convex hull using the pixel RGB values ​​included in the multi-point images and may form a color set with the vertices of the 3D convex hull. For example, the light source color included in the scene included in the image may be included in the color set. In one embodiment, the color set may be generated by operations 310 through 320 of FIG. 3. According to one embodiment, the electronic device (101) may perform learning on the 3D geometry and weights (e.g., weights of the color set) based on the input image. In operation 510, the electronic device (101) according to one embodiment may select a first color from the color set as a light source candidate. The electronic device (101) may select any color within the color set as a light source candidate. The electronic device (101) can select light source candidate colors within a color set, excluding colors that have already been identified as not being light sources. If all colors within the color set are designated as light source candidates and no light source is found during the completion of the next 520 to 545 operations, it can be determined that the input image does not contain a light source.

[0091] In operation 520, an electronic device (101) according to one embodiment can learn the weights of the colors included in the color set and the unique colors in three-dimensional space through volume rendering. The electronic device (101) can relearn the weights of the color set to remove the light source effect for the remaining colors excluding the first color in the color set. The electronic device (101) can set the weights of the remaining colors excluding the first color in the color set to have discrete values ​​during the learning process. The electronic device (101) can perform learning on the weights of the first color as continuous values. The electronic device (101) can reflect the unique color of each object for the first color. For example, the electronic device (101) accumulates colors other than the light source color only by multiplying the color of the color set by the weight, but the light source color can be calculated by multiplying the color of the color set by the weight and the unique color of the object together. The electronic device (101) learns different weights for the color of the light source (e.g., the first color which is a candidate color of the light source) and the color that is not the color of the light source among the colors included in the color set, and can separate the influence of the light source and the influence of the surrounding environment by reflecting the object's inherent color in the case of the color of the light source.

[0092] In operation 530, an electronic device (101) according to one embodiment can learn a model that predicts scene components of a three-dimensional structure through physically based rendering. Scene components may include emission, material (albedo, roughness), and light from the surrounding environment (environment map) in three-dimensional space. The electronic device (101) can perform physically based rendering of how the light will be reflected into the camera based on the scene components of the three-dimensional structure. The light received at each three-dimensional location is determined by the emission and light from the surrounding environment, and the amount of light reflected into the camera can be calculated using the material. The electronic device (101) can repeat the learning until the reflected light resulting from the physically based rendering looks the same as the learning image (input image), that is, until the difference between the two is minimized. The electronic device (101) can learn to have the reflection calculated from the light source based on the influence of the separated light source equal to the influence of the light source, and the reflection calculated from the surrounding environment equal to the influence of the surrounding environment by learning the weights for the light source color and the remaining colors differently in the set of colors.

[0093] In operation 540, an electronic device (101) according to one embodiment can determine whether the error between the physical-based rendering result and the training image is greater than a predetermined first threshold. If the electronic device (101) determines that the error between the rendering result and the training image is greater than the first threshold, it can identify that the first color set as the light source candidate color is not the color of the light source (operation 545). If the error between the physical-based rendering result and the training image falls within the predetermined first threshold, the electronic device (101) can identify that the first color set as the light source candidate color is the color of the light source (operation 550).

[0094] In operation 545, the electronic device (101) can identify that the first color is not the color of the light source if the error between the rendering result and the training image is greater than the first threshold. The electronic device (101) can select the next light source candidate within the color set. The electronic device (101) can select the second color as the light source candidate from the remaining colors excluding the colors identified as not being the color of the light source (e.g., the first color) in the color set. The electronic device (101) can repeat operations 520 through 540 to verify whether the next light source candidate color (e.g., the second color) is the color of the light source. If the electronic device (101) verifies that the second color is also not the color of the light source, it can select the next color from the color set again as the light source candidate color. In this way, the electronic device (101) can verify whether all colors in the color set are the color of the light source by repeating operations 520 through 545 until the color of the light source is found.

[0095] In operation 550, an electronic device (101) according to one embodiment can extract material and light source information from a three-dimensional structure when a light source color is identified. The electronic device (101) can restore material information and light source information through a learning model for scene components of a three-dimensional structure. The electronic device (101) can render a three-dimensional space based on light source information and material information for the three-dimensional structure. The electronic device (101) can receive new material or light source information for the three-dimensional space from a user. The electronic device (101) can perform fine-tuning according to the input new material or light source information through physically based rendering. The electronic device (101) can output an edited image by performing volume rendering for the three-dimensional scene modified by user input. For example, the electronic device (101) can receive user input to change the material of an object and modify the three-dimensional space by considering the influence of the light source on the changed material through physically based rendering. Alternatively, the electronic device (101) may receive user input that changes the position, size, intensity, or color of a light source and modify the three-dimensional space by considering the influence of the changed light source through physically based rendering. The electronic device (101) may perform volume rendering of the modified three-dimensional space according to the user input and output a 2D image of a first viewpoint of the three-dimensional space. The electronic device (101) may provide a UI for changing the viewpoint of the three-dimensional space, and the user may change the viewpoint of the three-dimensional space to view a 2D image of the desired viewpoint.

[0096] FIGS. 6a and 6b are flowcharts illustrating detailed operations of an electronic device restoring a light source according to one embodiment of the present disclosure.

[0097] Each operation of FIG. 5 can be executed as a more specific operation, such as the operations of FIG. 6a and 6b. Each operation of FIG. 6a and 6b may correspond to a part of each operation of FIG. 5. For example, operation 601 may correspond to operation 510, and operations 602 to 603 may correspond to operation 520. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0098] In operation 601, an electronic device (101) according to one embodiment may select a light source candidate color from among the colors included in a color set. The electronic device (101) may assume a light source candidate color and then verify whether the light source candidate color corresponds to a light source through a learning process. Since the color set includes all vertices of the convex polygon of the pixel RGB values ​​included in the input images, the color of a light source with strong light intensity may be included. The electronic device (101) may select any one of the colors included in the color set. During the iteration of learning, the electronic device (101) may exclude colors from the color set that have already been verified not to be the color of a light source and select the next light source candidate color.

[0099] In operation 602, an electronic device (101) according to one embodiment may define and initialize discrete weights that each set of colors may have. A weight is an indicator of how important each color in a set of colors is. The weights may be updated to discrete weight values ​​that are appropriate for the scene during the learning process. Among the colors included in the set of colors, light sources may have continuous weights, but non-light source colors may have discrete weights. In operation 601, the remaining colors, excluding the designated light source candidate colors, may be determined to have any one of the discrete weights during the learning process.

[0100] In operation 603, an electronic device (101) according to one embodiment can predict a unique color in 3D space. The electronic device (101) can predict the color of an object (the object's unique color) that is not a light source based on the weights of a set of colors using a learning model.

[0101] In operation 604, an electronic device (101) according to one embodiment may select the nearest discrete weight and perform volume rendering. The electronic device (101) may replace the weights of a set of colors with the nearest value among predefined fixed values ​​(discrete weights). However, the electronic device (101) may discretely replace the weights only for the remaining colors excluding the light source candidate color. The electronic device (101) may allow the weights of the light source candidate color to have continuous values.

[0102] The electronic device (101) can calculate the radiance in 3D space using weights of a discretized set of colors and perform volume rendering. The electronic device (101) can render a scene in 3D using the color of a light source, the weights of a set of colors, and the intrinsic color of an object. The color of the light source can be considered as light reflected from an object.

[0103] In operation 605, the electronic device (101) according to one embodiment can determine whether the difference between the volume rendering result and the input image is minimized. The electronic device (101) can repeat operations 602 through 604 until the difference between the volume rendering result and the input image is minimized, that is, until the volume rendering result and the input image are equal within a predetermined error. The electronic device (101) can learn the weights of the color set so that the volume rendering result and the input image are equal. In operation 604, the electronic device (101) can learn so that the amount of change of the weights of the color set is reduced. The electronic device (101) can learn so that the weights of the color set become equal to the discrete weights, while simultaneously learning so that the discrete weights become equal to the average of the weights of the color set. The electronic device (101) can distinguish the light entering the camera from the volume rendering result into 1) light that starts from a light source and is visible directly or through reflection, and 2) light that starts from the surroundings (environment map) and is visible through reflection.

[0104] In operation 606, an electronic device (101) according to one embodiment may initialize a surface property prediction model and an albedo prediction model in 3D space. The surface property prediction model in 3D space may be defined and initialized to predict, for example, the degree of emission of an object, material (albedo, roughness), and light in the environment (environment map). The albedo prediction model may be copied and initialized from the model that predicted the object's intrinsic color in operation 603 (operation 608) and fine-tuned.

[0105] In operation 607, an electronic device (101) according to one embodiment can predict the properties of a scene using a surface property prediction model. The electronic device (101) can calculate the components of each scene using initialized prediction models.

[0106] In operation 609, an electronic device (101) according to one embodiment can perform physically based rendering of how light is reflected into the camera based on scene components. The light received at each three-dimensional location can be determined by the emission and the surrounding environment map. The electronic device (101) can calculate how much is reflected into the camera using surface color (albedo, roughness). Since the electronic device (101) has separated whether the light entering the camera is light from a light source or light from the surrounding environment through operation 605, in operation 609, it can calculate the reflections from the light source and the surrounding environment separately based on this.

[0107] In operation 610, the electronic device (101) according to one embodiment can determine whether the difference between the learning result, the physically based rendering result, and the input image is minimized. The electronic device (101) can repeat operation 609 to learn until the physically based rendering result becomes equal to the input image. The electronic device (101) can learn that the reflection calculated from the light source is equal to the influence of the light source, and that the reflection calculated from the surrounding environment is equal to the influence of the surrounding environment.

[0108] In operation 611, an electronic device (101) according to one embodiment can determine whether the error between the physically based rendering result and the training image is smaller than a predetermined first threshold. If the rendering result error is smaller than the first threshold, the electronic device (101) can identify that the light source candidate color selected in operation 601 is the color of the light source (operation 612). If the rendering result error is greater than or equal to the first threshold, the electronic device (101) can identify that the light source candidate color selected in operation 601 is not the color of the light source. The electronic device (101) can perform operation 601 again to determine whether another color among the color set is a light source. To perform a verification process (operations 602 to 612) for the new light source candidate color, the electronic device (101) can initialize an albedo prediction model with a unique color prediction model (operation 613).

[0109] The electronic device (101) can verify whether each color of the color set is a color of the light source by repeating operations 601 to 613, and can predict the range of influence of the light source using prediction models.

[0110] FIG. 7 is a flowchart illustrating the operation of an electronic device modifying a light source of a rendering screen according to user input, according to one embodiment of the present disclosure.

[0111] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1) can reconstruct a 3D space from 2D images including a light source and provide a rendering screen to a user. The electronic device (101) can output a modified rendering screen as the user modifies the light source (e.g., lighting) on ​​the rendering screen. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0112] In operation 710, an electronic device (101) according to one embodiment may receive an image including lighting. The electronic device (101) may, for example, use a camera to acquire multi-view images of a specific space or object with lighting.

[0113] In operation 720, an electronic device (101) according to one embodiment can output a first viewpoint image by emphasizing lighting through 3D space rendering. The electronic device (101) can construct a 3D space for the image, restore a light source (lighting), and identify an area of ​​influence by the lighting. In one embodiment, the electronic device (101) can construct a 3D space and perform light source restoration according to a rendering operation including light source restoration of FIG. 3.

[0114] The electronic device (101) can render a 3D space to generate an image for a first viewpoint. The first viewpoint may be any viewpoint in the 3D space. The initial output rendering screen may be a preset viewpoint (e.g., front). The electronic device (101) may change the viewpoint displayed by user input. For example, it may rotate the 3D space or zoom in or zoom out upon receiving touch and drag actions on the display screen. The electronic device (101) may output a rendering screen for the viewpoint changed by user input.

[0115] The electronic device (101) can provide an emphasis effect on the lighting in 3D space. For example, it can make the borders bold or give a sparkling effect.

[0116] In operation 730, an electronic device (101) according to one embodiment may receive user input for lighting editing. The electronic device (101) may change lighting contained in 3D space by receiving user input for a rendering image displayed on a display screen. User input for lighting editing may be, for example, changing at least one of the color, intensity, position, size, or shape of the lighting. In one embodiment, user input may be, for example, a touch, a long press, a double touch, a drag, a zoom-in touch, or a zoom-out touch. The user may move the position of the lighting through a long press and drag input on the lighting. The user may change the size of the lighting through a zoom-in or zoom-out input with two fingers after selecting the lighting. The user may change the shape of the lighting through a drag input after selecting the lighting.

[0117] In operation 740, an electronic device (101) according to one embodiment may output a scene reflecting edited lighting. Based on user input regarding lighting editing, the electronic device (101) may perform physically based rendering again, taking into account the influence of the light source. For example, if the user input is to change yellow lighting to orange lighting, the electronic device (101) may output a rendered screen reflecting the color of the changed lighting in 3D space. The electronic device (101) may output a portion of 3D space through a 2D display, but may output a 2D image corresponding to a 360-degree rotation of 3D space by user input (e.g., touch and drag input). The electronic device (101) may output an image for every pixel in 3D space according to user input.

[0118] FIGS. 8a and 8b are examples of lighting editing screens according to one embodiment of the present disclosure.

[0119] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1) provides an editing function for a rendering screen and can output a re-rendered result screen in real time in response to receiving an editing input for a light or object included in the rendering screen.

[0120] Referring to FIG. 8a, according to one embodiment, an electronic device (101) may display a rendering screen on a first screen (810) of a display and display lighting editing options on a second screen (820). For example, the electronic device (101) may highlight lighting on the rendering screen. For example, effects (e.g., halo, thick line) may be applied to the lighting border. The electronic device (101) may display an image of a first viewpoint of a 3D scene on the first screen (810), and the viewpoint of the 3D scene may be changed by a user touch or drag action. In response to the change in the viewpoint of the 3D scene, the electronic device (101) may output a rendering image of that viewpoint in real time on the first screen (810).

[0121] The second screen (820) may include an editing option that can change the color of the light or change the intensity of the light. The electronic device (101) can receive user input to select the color of the light or adjust the intensity when the editing option to be changed is selected for the light highlighted on the first screen (810).

[0122] Referring to FIG. 8b, according to one embodiment, an electronic device (101) may display a rendering screen on a third screen (830) of a display. The electronic device (101) may provide a function to intuitively edit lighting by touching the third screen (830). For example, a user may move the position of a light on the third screen (830) by long-pressing and dragging it. A user may change the size of a light by zooming out with two fingers. A user may select a light by touch gesture, and when changeable shape icons are displayed, select one of the shape icons to change the shape of the light. Based on user input executing the lighting editing function, the electronic device (101) may generate a rendering scene reflecting the lighting changes and display it as a fourth screen (840). The electronic device (101) may output a physically based rendered result so that the influence of the lighting on surrounding objects is reflected by the changed lighting.

[0123] According to one embodiment, the electronic device (101) can learn past user editing history data to recommend lighting editing or automatically reflect user preferences. For example, the electronic device (101) can automatically change the lighting color of the rendering screen to a lighting color preferred by the user and output a guidance message.

[0124] The embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" each may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0125] The term “module” as used in the embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0126] One embodiment of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0127] According to one embodiment, the method according to the embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0128] According to one embodiment, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to one embodiment, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to one embodiment, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device, camera; display; Memory comprising at least one storage medium for storing instructions; and It includes at least one processor including a processing circuit; When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Using the above camera, a plurality of two-dimensional images are obtained that are captured at a plurality of viewpoints of a first scene including a first light source, and Generating a color set including some of the multiple colors included in the plurality of two-dimensional images above, and Predicting the structure of the 3D space and the radiance of the first scene through volume rendering of the plurality of 2D images, and Select a first color from the above set of colors as a candidate color for the light source, and Predicting the weights of the color set and the object's intrinsic color at each pixel in the 3D space through volume rendering, Predicting scene components in the 3D space through physically based rendering of the plurality of 2D images, and If the error between the pixel values ​​of the first scene and the plurality of two-dimensional images in the above physically based rendering result is within a first threshold value, the first color is identified as the color of the first light source, and If the error between the pixel values ​​of the first scene and the plurality of two-dimensional images in the physically based rendering result is greater than the first threshold, a light source candidate color is re-selected from the set of colors, and An electronic device that causes to generate an image of a first viewpoint in the 3D space when the above-mentioned physical-based rendering is completed.

2. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that causes the above color set to include colors corresponding to each vertex of a three-dimensional polygon (convex hull) formed by connecting the outermost boundary points among the points corresponding to multiple colors included in the plurality of two-dimensional images in a color space composed of RGB values.

3. In Paragraph 1 or 2, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Among the multiple colors included in the above color set, the weights for colors other than the color designated as the light source candidate color are initialized and updated as discrete values, and An electronic device that causes the weights for the colors designated as the above light source candidate colors to be initialized and updated as continuous values.

4. In any one of paragraphs 1 through 3, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that, while predicting the weights of the color set through the volume rendering, causes the weights for colors other than the color designated as the light source candidate color to be updated by selecting the closest value among the determined discrete values.

5. In any one of paragraphs 1 through 4, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that causes volume rendering for predicting the weights of the color set and the intrinsic color of the object to be repeated until the difference between the result of the volume rendering and the plurality of two-dimensional images reaches within a predetermined error.

6. In any one of paragraphs 1 through 5, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that, while predicting the intrinsic color of the object through the volume rendering above, causes the color designated as the light source candidate color to reflect the color, weight of the color set, and the intrinsic color of the object.

7. In any one of paragraphs 1 through 6, An electronic device that enables the scene components of the above 3D space to include the degree of luminescence of an object, its material, and the light of the surrounding environment.

8. In any one of paragraphs 1 through 7, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that causes the above-described physically based rendering to predict scene components in the 3D space using a surface property prediction model and an albedo prediction model of the 3D space.

9. In any one of paragraphs 1 through 8, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that causes an emphasis effect to be displayed on the first light source of the image at the first time point and output through the display.

10. In any one of paragraphs 1 through 9, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Receiving touch or drag user input on the screen displaying the image at the first time point, and An electronic device that causes to output a rendering screen for a changed point in time based on the above touch or drag user input.

11. In any one of paragraphs 1 through 10, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Receiving user input regarding the first light source included in the image of the first time point output through the display, Reflecting light source properties changed by the above user input through physically based rendering, and Causing the above-mentioned physical-based rendering result image to be output through the above-mentioned display, and The above user input is an electronic device that changes at least one of the color, intensity, position, size, or shape of a light source.

12. In a method of an electronic device, The operation of acquiring a plurality of two-dimensional images captured at a plurality of viewpoints for a first scene including a first light source; The operation of generating a color set including some of the plurality of colors included in the plurality of two-dimensional images above; An operation of predicting the structure of the 3D space and the radiance of the first scene through volume rendering of the plurality of 2D images; The operation of selecting a first color among the above color set as a light source candidate color; An operation to predict the weight of the color set and the object's intrinsic color at each pixel in the 3D space through volume rendering; An operation of predicting scene components in the 3D space through physically based rendering of the plurality of 2D images; An operation to identify that the first color is the color of the first light source when the error between the pixel values ​​of the first scene and the plurality of two-dimensional images in the physically based rendering result is within a first threshold value; If the error between the pixel values ​​of the first scene and the plurality of two-dimensional images in the physically based rendering result is greater than the first threshold, the operation of returning to the operation of selecting a light source candidate color from the set of colors; and A method comprising the operation of generating an image of a first viewpoint in the 3D space when the above-mentioned physically based rendering is completed.

13. In Paragraph 12, The operation of generating a color set including some of the plurality of colors included in the plurality of two-dimensional images above is, A method comprising the operation of generating a color set including colors corresponding to each vertex of a three-dimensional polygon (convex hull) formed by connecting the outermost boundary points among the points corresponding to multiple colors included in the plurality of two-dimensional images in a color space composed of RGB values.

14. In Paragraph 12 or 13, The operation of predicting the weights of the color set and the object's intrinsic color at each pixel in the 3D space through the volume rendering above is, The operation of initializing and updating the weights for colors other than the color designated as the light source candidate color among the plurality of colors included in the above color set to discrete values; and A method comprising the operation of initializing and updating the weights for the colors designated as the light source candidate colors as continuous values.

15. A non-transitory computer-readable storage medium that records instructions, When the above instructions are executed individually or collectively by one or more processors, the one or more processors: The operation of acquiring a plurality of two-dimensional images captured at a plurality of viewpoints for a first scene including a first light source; The operation of generating a color set including some of the plurality of colors included in the plurality of two-dimensional images above; An operation of predicting the structure of the 3D space and the radiance of the first scene through volume rendering of the plurality of 2D images; The operation of selecting a first color among the above color set as a light source candidate color; An operation to predict the weight of the color set and the object's intrinsic color at each pixel in the 3D space through volume rendering; An operation of predicting scene components in the 3D space through physically based rendering of the plurality of 2D images; An operation to identify that the first color is the color of the first light source when the error between the pixel values ​​of the first scene and the plurality of two-dimensional images in the physically based rendering result is within a first threshold value; If the error between the pixel values ​​of the first scene and the plurality of two-dimensional images in the physically based rendering result is greater than the first threshold, the operation of returning to the operation of selecting a light source candidate color from the set of colors; and A storage medium that performs the operation of generating an image of a first viewpoint in the 3D space when the above-mentioned physical-based rendering is completed.

Images