Electronic device and method for reducing noise of image data in electronic device

Abstract

An electronic device according to an embodiment comprises: an image sensor; a memory including a first noise correction map and a gain of a first white balance; and at least one processor, wherein the at least one processor may: obtain image data from the image sensor; obtain the first noise correction map and the gain of the first white balance from the memory; generate a gain of a second white balance on the basis of the image data; generate a second noise correction map on the basis of the first noise correction map, the gain of the first white balance, and the gain of the second white balance; and generate an output image by applying the second noise correction map to the image data. Various other embodiments may be included.

Description

Electronic device and method for reducing noise in image data in an electronic device

[0001] The present disclosure relates to an electronic device and a method for reducing noise in image data in an electronic device.

[0002] Fixed pattern noise (FPN) occurs because the sensitivity and offset of each pixel in the image sensor differ slightly, resulting in different voltage outputs for each pixel even with the same amount of light. This is due to the non-uniform pixel characteristics of the image sensor, and can be further exacerbated by the differences in characteristics between the amplifier and the Analog to Digital Converter (ADC) used in the process of converting pixel values ​​into digital values.

[0003] Fixed pattern noise (FPN) can degrade image quality if not removed, as it always appears at the same location in consecutive images. While it can be removed using spatial filters, there may be limitations to effective removal as increasing filter intensity leads to a loss of image detail.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0005] An electronic device according to one embodiment includes an image sensor, a memory including a first noise correction map and a first white balance gain, and at least one processor. According to one embodiment, the at least one processor can acquire image data from the image sensor. The first noise correction map and the first white balance gain can be acquired from the memory. According to one embodiment, the at least one processor can generate a second white balance gain based on the image data. According to one embodiment, the at least one processor can generate a second noise correction map based on the first noise correction map, the first white balance gain, and the second white balance gain. According to one embodiment, the at least one processor can generate an output image by applying the second noise correction map to the image data.

[0006] A method for reducing noise in image data in an electronic device according to one embodiment may include an operation of acquiring image data from an image sensor of the electronic device. The method according to one embodiment may include an operation of acquiring the first noise correction map and the first white balance gain from the memory of the electronic device. The method according to one embodiment may include an operation of generating a second white balance gain based on the image data. The method according to one embodiment may include an operation of generating a second noise correction map based on the first noise correction map, the first white balance gain, and the second white balance gain. The method according to one embodiment may include an operation of generating an output image by applying the second noise correction map to the image data.

[0007] In a non-volatile storage medium storing instructions according to one embodiment, the instructions are configured to cause the electronic device to perform at least one operation when executed by the electronic device, wherein the at least one operation may include an operation of acquiring image data from an image sensor of the electronic device. An at least one operation according to one embodiment may include an operation of acquiring the first noise correction map and the first white balance gain from the memory of the electronic device. An at least one operation according to one embodiment may include an operation of generating a second white balance gain based on the image data. An at least one operation according to one embodiment may include an operation of generating a second noise correction map based on the first noise correction map, the first white balance gain, and the second white balance gain. An at least one operation according to one embodiment may include an operation of generating an output image by applying the second noise correction map to the image data.

[0008] FIG. 1 is a block diagram of an electronic device in a network environment according to one embodiment.

[0009] FIG. 2 is a block diagram of an electronic device according to one embodiment.

[0010] FIG. 3 is a drawing showing histograms of U-channel and V-channel according to lighting environment in one embodiment.

[0011] FIG. 4 is a flowchart illustrating the operation for generating noise correction information in an electronic device according to one embodiment.

[0012] FIG. 5 is a flowchart illustrating an operation for reducing image noise in an electronic device according to one embodiment.

[0013] FIG. 6 is a flowchart illustrating the operation of generating a second noise correction map in an electronic device according to one embodiment.

[0014] FIG. 7 is a diagram illustrating an operation for reducing noise in an image using noise correction information in an electronic device according to one embodiment.

[0015] FIGS. 8a and 8b are drawings comparing images to which a noise correction map has been applied in an electronic device according to one embodiment.

[0016] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to one embodiment. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through the server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0017] The processor (120) can control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., a program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0018] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0019] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0020] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0021] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0022] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0023] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0024] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0025] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0026] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0027] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0028] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0029] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0030] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0031] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0032] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0033] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0034] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0035] According to one embodiment, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0036] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0037] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0038] FIG. 2 is a block diagram of an electronic device according to one embodiment.

[0039] Referring to FIG. 2, according to one embodiment, an electronic device (201) (e.g., the electronic device (101) of FIG. 1) may include a processor (220), a memory (230), a camera (280), and a communication circuit (290).

[0040] According to one embodiment, the processor (220) can perform overall control operations of the electronic device (201). According to one embodiment, the processor (220) can control at least one other component (e.g., hardware or software component) of the electronic device (201) connected to the processor (220) by executing software (e.g., program (140) of FIG. 1), and can perform data processing or operations based on instructions. According to one embodiment, the instructions may include instructions composed of machine language that can be processed by the electronic device (201) or the processor (220). For example, the instructions may include instructions corresponding to operation instructions used in the program.

[0041] According to one embodiment, the processor (220) can check noise correction information (e.g., fixed pattern noise correction information) including at least one of a first white balance gain value or a first color temperature stored in memory (230) before shipment of the electronic device (201) and a first noise correction map (e.g., a first fixed pattern noise map).

[0042] According to one embodiment, the processor (220), in an operation to generate noise correction information in the electronic device (201) after shipment of the electronic device (201), may acquire noise correction information including at least one of a first white balance gain value or a first color temperature and a first noise correction map, and may store the acquired noise correction information (e.g., fixed pattern noise correction information) in memory (230).

[0043] According to one embodiment, in an operation to generate noise correction information, the processor (220) may acquire a plurality of image data from an image sensor of a camera (280) and generate a first noise correction map (e.g., a first fixed pattern noise map) based on pixel-by-pixel brightness values ​​of the acquired plurality of image data.

[0044] According to one embodiment, the processor (220) can generate a first noise correction map to reduce fixed pattern noise (FPN), which is a factor in image quality degradation if not removed, because it appears at the same location in continuous image data.

[0045] According to one embodiment, a first noise correction map for reducing fixed pattern noise (FPN) may include a noise correction map for reducing (removing) Dark Signal Non-Uniformity (DSNU), which is noise that causes an offset due to dark currents output from a plurality of pixels included in the image sensor of the camera (280) in a light-free state.

[0046] According to one embodiment, the processor (220) can continuously acquire multiple image data through the image sensor of the camera (280) in a lightless state by utilizing the characteristic that noise occurring at a specific location of multiple image data converges to a designated value even when the average value is calculated by accumulating the pixel values ​​of each image data, and then generate a noise correction map to reduce (remove) the DSNU (Dark Signal Non-Uniformity) using the calculated average value.

[0047] According to one embodiment, a first noise correction map for reducing fixed pattern noise (FPN) may include a noise correction map for reducing Pixel Response Non-Uniformity (PRNU), which is noise caused by a difference in the degree of response (sensitivity difference) of each of the plurality of pixels included in the image sensor of the camera (280) to light in the presence of light.

[0048] According to one embodiment, the processor (220) may include a process of estimating the residual image data, calculated as the difference between the image data acquired in a light-present state and the average image acquired through the camera (280), as noise. The processor may calculate the difference between the image acquired through the camera in a light-present state and the average image acquired through the camera, estimate the residual image calculated as the difference as noise, and then include a noise correction map to reduce the Pixel Response Non-Uniformity (PRNU) using the average value of the residual images calculated for a plurality of images.

[0049] According to one embodiment, the processor (220) may include at least one of a noise correction map for reducing (removing) Dark Signal Non-Uniformity (DSNU) or a noise correction map for reducing (removing) Pixel Response Non-Uniformity (PRNU) in the first noise correction map.

[0050] According to one embodiment, the processor (220) can generate a first noise correction map in the YUV domain (color space) to reduce fixed pattern noise (FPN) and can generate it individually for each YUV channel.

[0051] According to one embodiment, the processor (220), in the operation of generating noise correction information, acquires a plurality of image data through a camera (280) and can check the gain value of the first white balance applied to the acquired plurality of image data.

[0052] According to one embodiment, the processor (220) can perform a white balance function by receiving raw data (R, G, B) through the image sensor of the camera (280), analyzing the information of the acquired image data to determine the white balance gain values ​​(WBgainR, WBgainG, WBgainB) to be applied to each channel (R channel, G channel, B channel), calculating the color temperature of the ambient light source based on the determined white balance gain values, and applying (multiplying) the white balance gain values ​​(R Gain, G Gain, B Gain) to each channel (R channel, G channel, B channel) to correct the color temperature of the entire image.

[0053] According to one embodiment, the processor (220) can check the white balance gain value in the metadata stored in each of the plurality of image data.

[0054] According to one embodiment, the processor (220) acquires a plurality of image data from an image sensor of a camera (280), accumulates and stores white balance gain values ​​applied to each of the acquired plurality of image data (e.g., white balance gain values ​​of R, G, and B channels), and can identify a first white balance gain value through statistical processing of the stored white balance gain values ​​(e.g., average value, median value, minimum value, maximum value and / or average value by interval).

[0055] According to one embodiment, the processor (220) can accumulate the gain values ​​of the white balance sequentially verified together with the image data input sequentially, and can determine the average value calculated by dividing the sum of the accumulated white balance gain values ​​by the number of accumulated frames as the first white balance gain value.

[0056] According to one embodiment, the processor (220) acquires a plurality of image data through an image sensor of a camera (280), checks a white balance gain value applied to each of the acquired image data (e.g., a white balance gain value for each of the R, G, and B channels), checks a section containing the identified white balance gain value among a plurality of predefined gain value sections (e.g., a plurality of color temperature sections), and can identify the white balance gain value included in the identified section (range) as the first white balance gain value. According to one embodiment, in an operation to generate noise correction information, the processor (220) can calculate a first color temperature representing the color temperature of an ambient light source at the time when the plurality of image data is acquired from the image sensor of the camera (280) using the first white balance gain value.

[0057] According to one embodiment, the processor (220) can calculate a color temperature from the gain values ​​of the first white balance using the following <Equation 1> to <Equation 3> and confirm the calculated color temperature as the first color temperature. According to one embodiment, the processor (220) can calculate the tristimulus values ​​X, Y, and Z of the eye using the first white balance gain values ​​(WBgainR, WBgainG, WBgainB) and the pre-calculated WBgain2XYZ[0] ~ WBgain2XYZ[8] array according to the following <Equation 1>.

[0058] According to one embodiment, the processor (220) can calculate color coordinates x and y by substituting the tristimulus values ​​of the eye, X, Y, and Z, calculated in the following <Equation 1>, using the following <Equation 2>.

[0059] According to one embodiment, the processor (220) calculates a color temperature by substituting the color coordinates x and y calculated in the following <Equation 2> using the following <Equation 3>, and can confirm the calculated color temperature as the first color temperature.

[0060] <Formula 1>

[0061] X = WBgain2XYZ[0] * WBgainR + WBgain2XYZ [1] * WBgainG + WBgain2

[0062] Y = WBgain2XYZ[3] * WBgainR + WBgain2XYZ[4] * WBgainG + WBgain2XYZ[5] * WBgainB

[0063] Z = WBgain2XYZ[6] * WBgainR + WBgain2XYZ[7] * WBgainG + WBgain2XYZ[8] * WBgainB

[0064] <Equation 2>

[0065] x = X / (X + Y + Z)

[0066] y = Y / (X + Y + Z)

[0067] z = Z / (X + Y + Z)

[0068] (x + y + z = 1)

[0069] <Equation 3>

[0070] Color Temperature = ((((-437.0F * n) + 3601.0F) * n) - 6861.0F) * n + 5514.31F

[0071] n = (x - 0.3320F) / (y - 0.1858F)

[0072] According to one embodiment, when the processor (220) acquires image data from the image sensor of the camera (280), it can check noise correction information (e.g., fixed pattern noise information) including at least one of a first white balance gain value or a first color temperature stored in the memory (280) and a first noise correction map (e.g., fixed pattern noise correction map).

[0073] According to one embodiment, the processor (220) can generate a second noise correction map by applying a compensation value to the first noise correction map included in the noise correction information using the gain value of the first white balance included in the noise correction information, and generate an output image in which noise is reduced (removed) from the image data based on the second noise correction map.

[0074] According to one embodiment, the processor (220) can compare the lighting environment (e.g., color temperature of an ambient light source) where noise correction information (e.g., fixed pattern noise information) was generated with the lighting environment (e.g., color temperature of an ambient light source) at the time when image data was acquired from the image sensor of the camera (280) (e.g., color temperature of an ambient light source) where image data was acquired from the image sensor of the camera (280) when the lighting environment (e.g., color temperature of an ambient light source) where image data was acquired from the camera (280) is different from the lighting environment (e.g., color temperature of an ambient light source) where image data was acquired from the image sensor of the camera (280) where noise correction information (e.g., fixed pattern noise information) was generated, because if the image is overcorrected or miscorrected when applying the noise correction information (e.g., fixed pattern noise information) to the image acquired from the image sensor of the camera (280), the image may be overcorrected or miscorrected.

[0075] According to one embodiment, the processor (220) can calculate a second color temperature representing the color temperature of the ambient light source at the time when image data is acquired through the camera (280) in order to compare the first color temperature included in the noise correction information with the color temperature of the ambient light source at the time when image data is acquired through the camera (280).

[0076] According to one embodiment, the processor (220) can check the gain value of the second white balance applied to image data acquired through the image sensor of the camera (280) and calculate the second color temperature based on the checked gain value of the second white balance. Here, the gain value of the second white balance is a value calculated during the automatic white balance (AWB) processing process, and rather than separately calculating the color temperature anew, it can be implemented by utilizing the already calculated gain value to infer the color temperature or check the corresponding color temperature.

[0077] According to one embodiment, the processor (220) can check the gain value of the second white balance in the metadata of the image data.

[0078] According to one embodiment, the processor (220) can calculate the color temperature from the gain value of the second white balance using the above <Equation 1> to <Equation 3> and confirm the calculated color temperature as the second color temperature.

[0079] According to one embodiment, the processor (220) can calculate the tristimulus values ​​X, Y, and Z of the eye using the second white balance gain values ​​(WBgainR, WBgainG, WBgainB) and the pre-calculated WBgain2XYZ[0] ~ WBgain2XYZ[8] array according to the above <Equation 1>.

[0080] According to one embodiment, the processor (220) can calculate color coordinates x and y by substituting the tristimulus values ​​of the eye, X, Y, and Z, calculated in the above <Equation 1>, using the above <Equation 2>.

[0081] According to one embodiment, the processor (220) calculates a color temperature by substituting the color coordinates x and y calculated in the above <Equation 2> using the above <Equation 3>, and can confirm the calculated color temperature as a second color temperature.

[0082] According to one embodiment, the processor (220) can compare a first color temperature included in the noise correction information with a second color temperature calculated from image data obtained through the camera (280), and compare the difference between the first color temperature and the second color temperature with a threshold value.

[0083] According to one embodiment, the processor (220) determines a compensation value of 1 when the difference between the first color temperature and the second color temperature is less than a threshold value, performs a correction operation of applying a first noise correction map included in the noise correction information to the image data, and can generate an output image in which noise (e.g., fixed pattern noise (FPN)) is reduced (removed) from the image data through the correction operation.

[0084] According to one embodiment, the processor (220) can generate a second noise correction map by applying a compensation value to the first noise correction map to reduce noise in the image data when the difference between the first color temperature and the second color temperature is greater than or equal to a threshold value.

[0085] According to one embodiment, the processor (220) can calculate a compensation value using the gain value of the first white balance included in the noise correction information and the gain value of the second white balance applied to the image data.

[0086] According to one embodiment, the processor (220) can calculate a compensation value (WB Compensation Gain(R, G, B)) based on the gain value of the white balance for each R, G, and B channel using the following <Equation 4>.

[0087] <Equation 4>

[0088]

[0089] Here, Calibration WB Gain represents the gain value of the first white balance, and Correction WB Gain may represent the gain value of the second white balance.

[0090] According to one embodiment, the processor (220) may convert the first noise correction map from a non-linear domain to a linear domain to generate a second noise correction map by applying a compensation value to the first noise correction map in a domain (color space) where linearity is guaranteed, and then convert the second noise correction map from a linear domain to a non-linear domain to generate the second noise correction map by applying a compensation value to the first noise correction map converted to a linear domain.

[0091] According to one embodiment, the processor (220) can convert a first noise correction map from a YUV domain to an RGB domain (Domain Conversion), perform inverse gamma correction on the first noise correction map converted to the RGB domain (Inverse Gamma), apply an inverse color correction matrix (Inverse CCM (Color Correction Matrix)) to the first noise correction map that has undergone inverse gamma correction (Inverse CCM (Color Correction Matrix)), and apply a compensation value to the first noise correction map to which the inverse color correction matrix has been applied to generate a second noise correction map.

[0092] According to one embodiment, the processor (220) can apply a color correction matrix to a second noise correction map (Color Correction Matrix (CCM)), perform gamma correction on the second noise correction map to which the color correction matrix has been applied (Gamma), and convert the second noise correction map to which gamma correction has been performed from the RGB domain to the YUV domain.

[0093] According to one embodiment, the processor (220) performs a correction operation of applying a second noise correction map to image data, and through the correction operation, can generate an output image in which noise (e.g., Fixed Pattern Noise (FPN)) is reduced (removed) from the image data.

[0094] According to one embodiment, the processor (220) can generate an output image in which noise (e.g., Fixed Pattern Noise (FPN)) is reduced (removed) from image data acquired through the image sensor of the camera (280) based on the second noise correction map, and then delete the second noise correction map.

[0095] According to one embodiment, the processor (220) can check a plurality of noise correction maps stored in memory (230) before shipping the electronic device (201).

[0096] According to one embodiment, the processor (220) can divide the color temperature in memory (230) into sections (e.g., 3000K, 4000K, 5000K and / or 6000K) and check the optimal noise correction map (e.g., representative noise map) set for each section.

[0097] For example, the processor (220) can divide the color temperature into multiple intervals such as "0 - 1000K", "1001K - 3000K", and pre-set an optimal noise correction map (e.g., a representative noise map) for each interval.

[0098] According to one embodiment, the processor (220) sets an optimal noise correction map (e.g., a representative noise map) for each of the plurality of color temperature ranges, and can use the optimal noise correction map (e.g., a representative noise map) set for each range by interpolating in the case of an intermediate color temperature (Kelvin).

[0099] For example, if the processor (220) has saved a noise correction map corresponding to 1000K as "FPN Map_1" and a noise correction map corresponding to 3000K as "FPN Map_2", and if an image is taken in a 2000K environment, it can generate and apply a noise correction map optimized for 2000K (e.g., a representative correction map) by interpolating using "FPN Map_1" and "FPN Map_2".

[0100] According to one embodiment, the processor (220), in the operation of generating noise correction information in the electronic device (201) after shipment of the electronic device (201), may divide the color temperature into sections (e.g., 3000K, 4000K, 5000K and / or 6000K) and generate an optimal noise correction map (e.g., a representative correction map) for each section and store it in memory (230).

[0101] According to one embodiment, the memory (230) may be implemented substantially identically or similarly to the memory (130) of FIG. 1.

[0102] According to one embodiment, noise correction information including at least one of a first white balance gain value or a first color temperature and a first noise correction map may be stored in the memory (230).

[0103] According to one embodiment, a plurality of noise correction maps set for each color temperature range may be stored in the memory (230).

[0104] According to one embodiment, the camera (280) may be implemented substantially identically or similarly to the camera module (180) of FIG. 1.

[0105] According to one embodiment, the camera (280) can acquire an image through an image sensor.

[0106] According to one embodiment, the image sensor is a semiconductor device composed of a plurality of pixels that can detect light and convert it into an electronic signal.

[0107] According to one embodiment, the communication circuit (290) can form a communication connection with an external electronic device (e.g., another electronic device, or a server) using various types of communication methods and transmit and / or receive data. As described above, the communication methods may include communication methods that establish a direct communication connection, such as Bluetooth and Wi-Fi Direct, communication methods that use an access point (AP) (e.g., Wi-Fi communication), or communication methods that use cellular communication using a base station (e.g., 3G, 4G / LTE, 5G). Since the communication circuit (290) can be implemented as described above in the communication module (190) in FIG. 1, a redundant description is omitted.

[0108] FIG. 3 is a diagram showing histograms of U-channel and V-channel according to lighting environment according to one embodiment.

[0109] Figure 3 above shows the distribution (310) of the U-channel histogram and the distribution (330) of the V-channel histogram, where the horizontal axis of each graph represents the pixel brightness value of the noise correction map and the vertical axis represents the number of pixels having the corresponding pixel brightness value.

[0110] When looking at the distribution of the U-channel histogram (310) and the distribution of the V-channel histogram (330), it can be seen that the U-channel and V-channel have opposite tendencies depending on the color temperature. When the color temperature is high (Cal 5500k), it can be seen that the distribution of the number of pixels in the U-channel tends to decrease in the region of relatively high pixel brightness values, while the distribution of the number of pixels in the V-channel tends to increase in the region of relatively low pixel brightness values. When the color temperature is low (Cal 2700k), it can be seen that the distribution of the number of pixels in the U-channel tends to increase in the region of relatively high pixel brightness values, while the distribution of the number of pixels in the V-channel tends to decrease in the region of relatively low pixel brightness values.

[0111] For example, when the color temperature is low, the red (R) component is strong and the blue (B) component is weak; therefore, when converting from the RGB domain (color space) to the YUV domain (color space), the V channel value may become relatively larger due to the increase in the R channel value. Consequently, if the lighting environment in which the noise correction map was generated (e.g., the color temperature of the surrounding environment) differs from the lighting environment at the time the image data was acquired through the camera (e.g., the color temperature of the surrounding environment), applying the fixed noise correction map to the image data acquired through the camera may result in miscorrection, causing the noise not to be properly removed or errors caused by the noise to occur after correction.

[0112] An electronic device (101 of FIG. 1; 201 of FIG. 2) according to one embodiment may include a memory (130 of FIG. 1; 230 of FIG. 2) including a first noise correction map and a first white balance gain, and at least one processor (120 of FIG. 1; 220 of FIG. 2). According to one embodiment, the at least one processor may acquire image data from the image sensor. According to one embodiment, the at least one processor may acquire the first noise correction map and the first white balance gain from the memory. According to one embodiment, the at least one processor may generate a second white balance gain based on the image data. According to one embodiment, the at least one processor may generate a second noise correction map based on the first noise correction map, the first white balance gain, and the second white balance gain. According to one embodiment, the at least one processor can generate an output image by applying the second noise correction map to the image data.

[0113] According to one embodiment, the processor may obtain a first color temperature representing a color temperature associated with the first noise correction map from the memory. According to one embodiment, the processor may calculate a second color temperature representing the color temperature of the image data based on the gain of the second white balance. According to one embodiment, the processor may generate the second noise correction map based on a comparison of the first color temperature and the second color temperature.

[0114] According to one embodiment, the processor can determine a compensation value. According to one embodiment, the processor can generate the second noise correction map by applying the compensation value to the first noise correction map.

[0115] According to one embodiment, the processor may determine the compensation value based on the gain of the first white balance and the gain of the second white balance if the difference between the first color temperature and the second color temperature is greater than or equal to a threshold value. According to one embodiment, the processor may determine the compensation value to be 1 if the difference between the first color temperature and the second color temperature is less than the threshold value.

[0116] According to one embodiment, the processor can apply the second white balance gain to the output image.

[0117] In a processor according to one embodiment, the gain of the first white balance, the gain of the second white balance, the first noise correction map, and the second noise correction map may each include three values ​​for three color components. In a processor according to one embodiment, the first color temperature may include a value related to the first noise correction map. In a processor according to one embodiment, the second color temperature may include a value for the image data. In a processor according to one embodiment, the first noise correction map may include multiple values ​​for multiple pixels to reduce fixed pattern noise of the image sensor in images acquired from the image sensor. In a processor according to one embodiment, the second noise correction map may include multiple values ​​for multiple pixels to reduce fixed pattern noise of the image sensor in the image data.

[0118] According to one embodiment, the processor can convert the first noise correction map from a non-linear domain to a linear domain. According to one embodiment, the processor can generate the second noise correction map by applying the compensation value to the first noise correction map converted to the linear domain. According to one embodiment, the processor can convert the second noise correction map from a linear domain to a non-linear domain.

[0119] According to one embodiment, the processor may acquire a plurality of image data from the image sensor and generate the first noise correction map based on the pixel-by-pixel brightness value of each of the plurality of image data. According to one embodiment, the processor may check the gain of the first white balance applied to the plurality of image data. According to one embodiment, the processor may calculate the first color temperature using the first white balance gain. According to one embodiment, the processor may store the first noise correction map, the gain of the first white balance, and the first color temperature in the memory.

[0120] According to one embodiment, the processor may delete the second noise correction map after generating the output image.

[0121] FIG. 4 is a flowchart illustrating operations for generating noise correction information in an electronic device according to one embodiment. Operations for generating noise correction information may include operations 401 through 409. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, at least two operations may be performed in parallel, or other operations may be added.

[0122] According to one embodiment, the operation for generating noise correction information in an electronic device may be performed before or after shipment of the electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2).

[0123] In operation 401, an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can acquire a plurality of image data through an image sensor of a camera (e.g., the camera (280) of FIG. 2).

[0124] In operation 403, an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can generate a first noise correction map (e.g., a fixed pattern noise map) based on pixel-by-pixel brightness values ​​of a plurality of image data.

[0125] According to one embodiment, the electronic device can generate a first noise correction map to reduce fixed pattern noise (FPN) that occurs at the same location in consecutive images and remains unremoved, thereby degrading the image data quality.

[0126] According to one embodiment, a first noise correction map for reducing fixed pattern noise (FPN) may include a noise correction map for reducing (removing) Dark Signal Non-Uniformity (DSNU), which is noise that causes an offset due to dark currents output from a plurality of pixels included in the image sensor of the camera (280) in a light-free state.

[0127] According to one embodiment, an electronic device can continuously acquire multiple image data through a camera in a lightless state, calculate the average value of pixel values ​​measured in each image data, and then generate a noise correction map to reduce (remove) the Dark Signal Non-Uniformity (DSNU) using the calculated average value, by utilizing the characteristic that noise occurring at a specific location in a plurality of image data converges to a specified value even when the average value is calculated by accumulating the pixel values ​​of each image.

[0128] According to one embodiment, a first noise correction map for reducing fixed pattern noise (FPN) may include a noise correction map for reducing Pixel Response Non-Uniformity (PRNU), which is noise caused by a difference in the degree of response (sensitivity difference) of each of the plurality of pixels included in the image sensor of a camera in the presence of light.

[0129] According to one embodiment, the electronic device may include a process of estimating the residual image data, calculated as the difference between the image data acquired in the presence of light and the average image acquired through the camera (280) in the presence of light, as noise. The device may include a noise correction map for reducing the Pixel Response Non-Uniformity (PRNU) by calculating the difference between the image acquired through the camera and the image acquired through the camera, estimating the residual image calculated as the difference as noise, and then using the average value of the residual images calculated for a plurality of images.

[0130] According to one embodiment, the electronic device may include at least one of a noise correction map for reducing (removing) Dark Signal Non-Uniformity (DSNU) or a noise correction map for reducing (removing) Pixel Response Non-Uniformity (PRNU) in the first noise correction map.

[0131] According to one embodiment, the electronic device can generate a first noise correction map in the YUV domain (color space) to reduce fixed pattern noise (FPN) and can generate it individually for each YUV channel.

[0132] In operation 405, an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can check the gain value of the first white balance applied to a plurality of image data.

[0133] According to one embodiment, when an electronic device receives raw image data (R, G, B) through an image sensor of a camera (e.g., camera (280) of FIG. 2), it analyzes the information of the acquired image data to determine white balance gain values ​​(WBgainR, WBgainG, WBgainB) to be applied to each channel (R channel, G channel, B channel), and can check the color temperature of an ambient light source based on the determined white balance gain values. According to one embodiment, the electronic device can perform a white balance function that corrects the color temperature of the entire image by applying (multiplying) the determined white balance gain values ​​(R Gain, G Gain, B Gain) to each channel (R channel, G channel, B channel). According to one embodiment, the electronic device can check the white balance gain values ​​in the metadata stored in each of the plurality of image data.

[0134] According to one embodiment, the electronic device acquires a plurality of image data through a camera, accumulates and stores white balance gain values ​​applied to each of the acquired plurality of image data (e.g., white balance gain values ​​of R, G, and B channels), and can identify a first white balance gain value through statistical processing of the stored white balance gain values ​​(e.g., average value, median value, minimum value, maximum value and / or average value by interval).

[0135] According to one embodiment, the electronic device can accumulate gain values ​​of white balance sequentially verified together with sequentially input image data, and can determine the average value calculated by dividing the sum of the accumulated gain values ​​of white balance by the number of accumulated frames as the first white balance gain value.

[0136] According to one embodiment, the electronic device acquires a plurality of image data through an image sensor of a camera, checks a white balance gain value applied to each of the acquired plurality of image data (e.g., a white balance gain value for each of the R, G, and B channels), checks a section containing the identified white balance gain value among a plurality of predefined gain value sections (e.g., a plurality of color temperature sections), and can identify the white balance gain value included in the identified section (range) as the first white balance gain value.

[0137] In operation 407, an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can determine a first color temperature representing the color temperature of an ambient light source at the time when a plurality of image data is acquired through an image sensor of a camera (e.g., the camera (280) of FIG. 2) by using the gain value of the first white balance.

[0138] According to one embodiment, the electronic device can calculate a first color temperature representing the color temperature of an ambient light source at the time when a plurality of images are acquired through a camera (e.g., camera (280) of FIG. 2) by using the gain value of the first white balance.

[0139] According to one embodiment, the electronic device can detect a first color temperature corresponding to a gain value of the first white balance among a plurality of color temperatures stored in the memory of the electronic device (e.g., memory (230) of FIG. 2).

[0140] According to one embodiment, the electronic device can calculate a color temperature from the gain value of the first white balance using the above <Equation 1> to the above <Equation 3>, and confirm the calculated color temperature as the first color temperature.

[0141] According to one embodiment, the electronic device can calculate the tristimulus values ​​X, Y, and Z of the eye using the first white balance gain values ​​(WBgainR, WBgainG, WBgainB) according to <Equation 1> and the pre-calculated array WBgain2XYZ[0] ~ WBgain2XYZ[8].

[0142] According to one embodiment, the electronic device can calculate color coordinates x and y by substituting the tristimulus values ​​of the eye, X, Y, and Z, calculated in <Equation 1> using <Equation 2>.

[0143] According to one embodiment, the electronic device can calculate a color temperature by substituting the color coordinates x and y calculated in <Equation 2> using <Equation 3>, and can confirm the calculated color temperature as the first color temperature.

[0144] In operation 409, an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) may store noise correction information including at least one of a first white balance gain value or a first color temperature and a first noise correction map.

[0145] According to one embodiment, the electronic device may store noise correction information (e.g., fixed pattern noise correction information) including at least one of a first white balance gain value or a first color temperature and a first noise correction map in a memory (e.g., memory (230) of FIG. 2).

[0146] FIG. 5 is a flowchart illustrating operations for reducing image noise in an electronic device according to one embodiment. Operations for reducing image noise may include operations 501 to 515. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, at least two operations may be performed in parallel, or other operations may be added.

[0147] In operation 501, an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can acquire image data through an image sensor of a camera (e.g., the camera (280) of FIG. 2).

[0148] In operation 503, an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can check noise correction information (e.g., fixed pattern noise correction information) including at least one of a first white balance gain value or a first color temperature and a first noise correction map stored in a memory (e.g., memory (230) of FIG. 2).

[0149] In operation 505, an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can check the gain value of the second white balance applied to image data acquired through the image sensor of a camera (e.g., the camera (280) of FIG. 2).

[0150] According to one embodiment, the electronic device can check the gain value of the second white balance in the meta-information of the image data.

[0151] In operation 507, an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can determine a second color temperature representing the color temperature of an ambient light source at the time when image data is acquired through an image sensor of a camera (e.g., the camera (280) of FIG. 2) using the gain value of the second white balance.

[0152] According to one embodiment, the electronic device can calculate a color temperature from the gain value of the second white balance using the above <Equation 1> to <Equation 3>, and confirm the calculated color temperature as the second color temperature.

[0153] According to one embodiment, the electronic device can calculate the tristimulus values ​​X, Y, and Z of the eye using the second white balance gain values ​​(WBgainR, WBgainG, WBgainB) and the pre-calculated array WBgain2XYZ[0] ~ WBgain2XYZ[8] according to the above <Equation 1>.

[0154] According to one embodiment, the electronic device can calculate color coordinates x and y by substituting the tristimulus values ​​of the eye, X, Y, and Z, calculated in <Equation 1> using <Equation 2>.

[0155] According to one embodiment, the electronic device calculates a color temperature by substituting the color coordinates x and y calculated in <Equation 2> using <Equation 3>, and can confirm the calculated color temperature as a second color temperature.

[0156] In operation 509, an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can compare the difference between the first color temperature and the second color temperature with a threshold value.

[0157] According to one embodiment, the electronic device can compare a first color temperature included in noise correction information with a second color temperature calculated from image data obtained through an image sensor of a camera (e.g., camera (280) of FIG. 2), and compare the difference between the first color temperature and the second color temperature with a threshold value.

[0158] In operation 509, an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can generate an output image with noise reduced in image data based on a first noise correction map (e.g., a first fixed pattern noise correction map) in operation 511 if the difference between the first color temperature and the second color temperature is less than a threshold value.

[0159] According to one embodiment, the electronic device determines a compensation value to 1, performs a correction operation in which a first noise correction map included in the noise correction information is applied to image data, and generates an output image in which noise (e.g., Fixed Pattern Noise (FPN)) is reduced (removed) from the image data through the correction operation.

[0160] In operation 509, if the difference between the first color temperature and the second color temperature is greater than or equal to a threshold value, the electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can, in operation 513, generate a second noise correction map (e.g., a second fixed pattern noise correction map) by applying a compensation value to the first noise correction map.

[0161] According to one embodiment, the electronic device can calculate a compensation value using a first white balance gain value included in noise correction information and a second white balance gain value applied to image data.

[0162] According to one embodiment, the electronic device can calculate a compensation value (WB Compensation Gain(R, G, B)) based on the gain value of the white balance for each R, G, and B channel using the above <Equation 4>.

[0163] According to one embodiment, in order to generate a second noise correction map by applying a compensation value to a first noise correction map in a domain (color space) where linearity is guaranteed, the electronic device may convert the first noise correction map from a non-linear domain to a linear domain, generate a second noise correction map by applying a compensation value to the first noise correction map converted to a linear domain, and convert the second noise correction map from a linear domain to a non-linear domain.

[0164] In operation 515, an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can generate an output image with noise reduced in image data based on a second noise correction map.

[0165] According to one embodiment, the electronic device performs a correction operation of applying a second noise correction map to image data, and can generate an output image in which noise (e.g., Fixed Pattern Noise (FPN)) is reduced (removed) from the image data through the correction operation.

[0166] According to one embodiment, the electronic device may generate an output image in which noise (e.g., fixed pattern noise (FPN)) is reduced (removed) from image data acquired through an image sensor of a camera (e.g., camera (280) of FIG. 2) based on a second noise correction map, and then delete the second noise correction map.

[0167] FIG. 6 is a flowchart illustrating the operation of generating a second noise correction map in an electronic device according to one embodiment. The operations for generating the second noise correction map may include operations 601 through 615. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, at least two operations may be performed in parallel, or other operations may be added.

[0168] In operation 601, an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can check a first noise correction map (e.g., a first fixed pattern noise correction map).

[0169] An electronic device according to one embodiment can confirm the start of generating a second noise correction map by applying a compensation value to a first noise correction map to reduce noise in an image.

[0170] In operation 603, an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can transform the domain of the first noise correction map.

[0171] According to one embodiment, the electronic device can convert a first noise correction map from a non-linear domain to a linear domain.

[0172] According to one embodiment, the electronic device can convert a first noise correction map from the YUV domain to the RGB domain.

[0173] In operation 605, an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can perform inverse gamma correction on the first noise correction map.

[0174] According to one embodiment, the electronic device can perform inverse gamma correction on a first noise correction map converted to the RGB domain.

[0175] In operation 607, an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) may apply an inverse color conversion matrix to a first noise correction map.

[0176] According to one embodiment, the electronic device can apply an inverse color conversion matrix to a first noise correction map that has undergone inverse gamma correction.

[0177] In operation 609, an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) may generate a second noise correction map (e.g., a second fixed pattern noise correction map) by applying a compensation value to a first noise correction map.

[0178] In operation 611, an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) may apply a color conversion matrix to a second noise correction map.

[0179] In operation 613, an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can perform gamma correction on a second noise correction map.

[0180] According to one embodiment, the electronic device can perform gamma correction on a second noise correction map to which a color conversion matrix has been applied.

[0181] In operation 615, an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can transform the domain of the second noise correction map.

[0182] According to one embodiment, the electronic device can convert a second noise correction map, on which gamma correction has been performed, from a linear domain to a non-linear domain.

[0183] According to one embodiment, the electronic device can convert the second noise correction map from the RGB domain to the YUV domain.

[0184] FIG. 7 is a diagram illustrating an operation for reducing noise in an image using noise correction information in an electronic device according to one embodiment.

[0185] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) may include a first noise correction information generating unit (711), a first white balance gain generating unit (713), and a first color temperature calculating unit (715) for generating noise correction information.

[0186] According to one embodiment, a first noise correction generating unit (711), a first white balance gain generating unit (713), and a first color temperature calculating unit (715) for generating noise correction information may be included in a processor (e.g., the processor (220) of FIG. 2) or configured separately.

[0187] According to one embodiment, a processor (e.g., processor (220) of FIG. 2) can perform the same functions as a first noise correction information generating unit (711), a first white balance gain generating unit (713), and a first color temperature calculating unit (715).

[0188] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) may include a second white balance gain generation unit (731), a second color temperature calculation unit (733), a color temperature comparison unit (735), a second noise correction generation unit (737), and a noise correction application unit (739) for reducing noise in an image.

[0189] According to one embodiment, for reducing noise in an image, a second white balance gain generation unit (731), a second color temperature calculation unit (733), a color temperature comparison unit (735), a second noise correction generation unit (737), and a noise correction application unit (739) may be included in a processor (e.g., the processor (220) of FIG. 2) or configured separately.

[0190] According to one embodiment, a processor (e.g., processor (220) of FIG. 2) can perform the same functions as a second white balance gain generation unit (731), a second color temperature calculation unit (733), a color temperature comparison unit (735), a second noise correction generation unit (737), and a noise correction application unit (739) for reducing noise in an image.

[0191] The operation of generating noise correction information according to one embodiment is as follows.

[0192] According to one embodiment, the first noise correction generation unit (711) can generate a first noise correction map (758a) based on pixel-by-pixel brightness values ​​of a plurality of image data (751) obtained through an image sensor of a camera (e.g., camera (280) of FIG. 2).

[0193] According to one embodiment, the first noise correction generation unit (711) generates the first noise correction map (758a) in the YUV domain (color space) for each YUV channel (FPN map ch1 , FPN map ch2 and FPN map ch3 It can be generated as )(758a).

[0194] According to one embodiment, the first white balance gain generation unit (713) can check the first white balance gain value (755) applied to a plurality of image data.

[0195] According to one embodiment, the first white balance gain generation unit (713) checks the first white balance gain value (755) based on the metadata of the image data, and the first white balance gain value (WBG_cal) applied to each channel (R channel, G channel, B channel) ch1 , WBG_cal ch2 , WBG_cal ch3 )(755) can be determined.

[0196] According to one embodiment, the first color temperature calculation unit (715) has a first white balance gain value (WBG_cal ch1 , WBG_cal ch2 , WBG_cal ch3 The first color temperature (CT_cal) (757) can be calculated using )(755).

[0197] According to one embodiment, a processor (e.g., processor (220) of FIG. 2) may store noise correction information in memory (230) including at least one of a first white balance gain value (755) or a first color temperature (757) and a first noise correction map (758a).

[0198] The operation for reducing image noise according to one embodiment is as follows.

[0199] According to one embodiment, the second white balance gain generation unit (731), upon acquiring image data (771) through an image sensor of a camera (e.g., camera (280) of FIG. 2), checks the second white balance gain value (773) based on the metadata of the image data, and the second white balance gain value (WBG_co) applied to each channel (R channel, G channel, B channel). ch1 , WBG_co ch2 , WBG_co ch3 )(773) can be determined.

[0200] According to one embodiment, the second color temperature calculation unit (733) has a gain value (WBG_co) of the second white balance. ch1 , WBG_co ch2 , WBG_co ch3 The second color temperature (CT_co) (775) can be calculated using )(773).

[0201] According to one embodiment, the color temperature comparison unit (735) compares the first color temperature (CT_cal) (757) and the second color temperature (CT_co) (775) of the noise correction information stored in the memory (230), and if it determines that the difference between the first color temperature (CT_cal) (757) and the second color temperature (CT_co) (775) is less than a threshold value, it can transmit first information indicating that the difference between the first color temperature (CT_cal) (757) and the second color temperature (CT_co) (775) is less than a threshold value to the noise correction application unit (739).

[0202] According to one embodiment, the noise correction application unit (739) confirms the reception of the first information and can correct the image data (771) based on the first noise map (758a) of the noise correction information stored in the memory (230) to generate an output image (759) with reduced noise.

[0203] According to one embodiment, the color temperature comparison unit (735) compares the first color temperature (CT_cal) (757) and the second color temperature (CT_co) (775) of the noise correction information stored in the memory (230), and if it is determined that the difference between the first color temperature (CT_cal) (757) and the second color temperature (CT_co) (775) is greater than or equal to a threshold value, it can transmit second information indicating that the difference between the first color temperature (CT_cal) (757) and the second color temperature (CT_co) (775) is greater than or equal to a threshold value to the second noise correction generation unit (737).

[0204] According to one embodiment, the second noise correction generation unit (737), upon confirming the reception of the second information, [retrieves] the gain value (755) of the first white balance and the gain value (WBG_co) of the second white balance of the noise correction information stored in the memory (230). ch1 , WBG_co ch2 , WBG_co ch3 A compensation value is generated using )(773), and a second noise correction map (758b) is generated by applying the compensation value to the first noise correction map (758a), and the second noise correction map (758b) can be transmitted to the noise correction application unit (739).

[0205] According to one embodiment, the noise correction application unit (739) can correct the image data (771) based on the second noise map (758b) to generate an output image (759) with reduced noise.

[0206] FIGS. 8a and 8b are drawings comparing images to which a noise correction map has been applied in an electronic device according to one embodiment.

[0207] FIGS. 8a and 8b are drawings comparing an output image in which the first noise correction map is applied to image data and an output image in which the second noise correction map is applied to image data, based on the lighting environment (color temperature of the ambient light source) at the time the image data is acquired through the camera, when the lighting environment (color temperature of the ambient light source) at the time the first noise correction map is generated is different from the lighting environment (color temperature of the ambient light source) at the time the image data is acquired through the camera.

[0208] Referring to FIG. 8a, when a first noise correction map is applied to first image data acquired through a camera, a first output image (811) containing a certain pattern of noise is generated as sufficient correction operation for the noise is not performed. On the other hand, when a second noise correction map is applied to the same first image data, a first output image (813) can be generated in which even the fixed pattern of noise remaining in the first output image (811) is removed (reduced) by applying the second noise correction map, as a correction operation using an accurate correction value is performed in contrast to the first noise correction map.

[0209] Referring to FIG. 8b, when a first noise correction map is applied to second image data acquired through a camera, a second output image (831) containing a certain pattern of noise is generated as sufficient correction operation for the noise is not performed. On the other hand, when a second noise correction map is applied to the same second image data, a correction operation using an accurate correction value is performed in comparison with the first noise correction map, thereby generating a second output image (833) in which even the fixed pattern of noise remaining in the first output image (831) is removed (reduced) by the application of the second noise correction map.

[0210] A method for reducing noise in image data in an electronic device (101 in FIG. 1; 201 in FIG. 2) according to one embodiment may include an operation of acquiring image data from an image sensor of the electronic device. The method according to one embodiment may include an operation of acquiring the first noise correction map and the first white balance gain from the memory of the electronic device. The method according to one embodiment may include an operation of generating a second white balance gain based on the image data. The method according to one embodiment may include an operation of generating a second noise correction map based on the first noise correction map, the first white balance gain, and the second white balance gain. The method according to one embodiment may include an operation of generating an output image by applying the second noise correction map to the image data.

[0211] The method according to one embodiment may include an operation of obtaining a first color temperature representing a color temperature associated with the first noise correction map from the memory. The method according to one embodiment may further include an operation of calculating a second color temperature representing the color temperature of the image data based on the gain of the second white balance. In the method according to one embodiment, the second noise correction map may be generated based on a comparison of the first color temperature and the second color temperature.

[0212] The method according to one embodiment may further include an operation of determining a compensation value. In the method according to one embodiment, the second noise correction map may be generated by applying the compensation value to the first noise correction map.

[0213] The method according to one embodiment may include an operation in which, if the difference between the first color temperature and the second color temperature is greater than or equal to a threshold value, the compensation value is determined based on the gain of the first white balance and the gain of the second white balance. The method according to one embodiment may further include an operation in which, if the difference between the first color temperature and the second color temperature is less than the threshold value, the compensation value is determined to be 1.

[0214] The method according to one embodiment may further include the operation of applying the second white balance gain to the output image.

[0215] In the method according to one embodiment, the gain of the first white balance, the gain of the second white balance, the first noise correction map, and the second noise correction map may each include three values ​​for three color components. In the method according to one embodiment, the first color temperature may include a value associated with the first noise correction map. In the method according to one embodiment, the second color temperature may include a value for the image data. In the method according to one embodiment, the first noise correction map may include multiple values ​​for multiple pixels to reduce fixed pattern noise of the image sensor in images acquired from the image sensor. In the method according to one embodiment, the second noise correction map may include multiple values ​​for multiple pixels to reduce fixed pattern noise of the image sensor in the image data.

[0216] The method according to one embodiment may include an operation of converting the first noise correction map from a non-linear domain to a linear domain. The method according to one embodiment may include an operation of generating the second noise correction map by applying the compensation value to the first noise correction map converted to the linear domain. The method according to one embodiment may further include an operation of converting the second noise correction map from a linear domain to a non-linear domain.

[0217] The method according to one embodiment may include an operation of acquiring a plurality of image data from the image sensor. The method according to one embodiment may include an operation of generating the first noise correction map based on the pixel-by-pixel brightness value of each of the plurality of image data. The method according to one embodiment may include an operation of checking the gain of the first white balance applied to the plurality of image data. The method according to one embodiment may include an operation of calculating the first color temperature using the gain of the first white balance. The method according to one embodiment may further include an operation of storing the first noise correction map, the gain of the first white balance, and the first color temperature in the memory.

[0218] The method according to one embodiment may further include the operation of deleting the second noise correction map after generating the output image.

[0219] In a non-volatile storage medium storing instructions according to one embodiment, the instructions are configured to cause the electronic device to perform at least one operation when executed by the electronic device, wherein the at least one operation may include an operation of acquiring image data from an image sensor of the electronic device. An at least one operation according to one embodiment may include an operation of acquiring the first noise correction map and the first white balance gain from the memory of the electronic device. An at least one operation according to one embodiment may include an operation of generating a second white balance gain based on the image data. An at least one operation according to one embodiment may include an operation of generating a second noise correction map based on the first noise correction map, the first white balance gain, and the second white balance gain. An at least one operation according to one embodiment may include an operation of generating an output image by applying the second noise correction map to the image data.

[0220] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains.

[0221] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs.

[0222] The electronic device according to one embodiment disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. The electronic device according to the embodiment of this document is not limited to the aforementioned devices.

[0223] One embodiment of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, each of phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as “first,” “second,” or “first” or “second” may be used simply to distinguish a component from another component and do not limit the components in any other aspect (e.g., importance or order). Where any (e.g., first) component is referred to as “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicationally,” it means that said component may be connected to said other component directly (e.g., wired), wirelessly, or through a third component.

[0224] The term "module" as used in an embodiment of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to an embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0225] One embodiment of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101) or electronic device (201)). For example, a processor (e.g., processor (220)) of the machine (e.g., electronic device (201)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0226] According to one embodiment, the method according to one embodiment disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0227] According to one embodiment, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to one embodiment, one or more of the components or operations among the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to one embodiment, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (101 in FIG. 1; 201 in FIG. 2), Image sensor; A memory including a first noise correction map and a first white balance gain (130 in FIG. 1; 230 in FIG. 2); and It includes at least one processor (120 in FIG. 1; 220 in FIG. 2), and said at least one processor, image data is obtained from the above image sensor, and The first noise correction map and the gain of the first white balance are obtained from the memory, and Based on the above image data, generate a second white balance gain, and A second noise correction map is generated based on the first noise correction map, the gain of the first white balance, and the second white balance gain, and An electronic device that generates an output image by applying the second noise correction map to the image data.

2. In claim 1, the processor, A first color temperature representing the color temperature associated with the first noise correction map is obtained from the memory above, and Based on the gain of the second white balance above, a second color temperature representing the color temperature of the image data is calculated, and The electronic device, wherein the second noise correction map is generated based on a comparison of the first color temperature and the second color temperature.

3. In claim 1, the processor, Determine the reward value, and The electronic device, wherein the second noise correction map is generated by applying the compensation value to the first noise correction map.

4. In claim 3, the processor, If the difference between the first color temperature and the second color temperature is greater than or equal to a threshold value, the compensation value is determined based on the gain of the first white balance and the gain of the second white balance, and An electronic device that determines the compensation value to be 1 if the difference between the first color temperature and the second color temperature is less than the threshold value.

5. In claim 1, the processor, An electronic device that applies the second white balance gain to the output image.

6. In claim 1, the processor, The gain of the first white balance, the gain of the second white balance, the first noise correction map, and the second noise correction map each include three values ​​for three color components, and The first color temperature above includes a value related to the first noise correction map, and The above second color temperature includes a value for the image data, and The first noise correction map above includes a plurality of values ​​for a plurality of pixels to reduce fixed pattern noise of the image sensor in images acquired from the image sensor, and The above second noise correction map is an electronic device comprising multiple values ​​for multiple pixels to reduce fixed pattern noise of the image sensor in the image data.

7. In claim 1, the processor, The above first noise correction map is transformed from a non-linear domain to a linear domain, and The compensation value is applied to the first noise correction map converted into the linear domain to generate the second noise correction map, and An electronic device that converts the above second noise correction map from a linear domain to a non-linear domain.

8. In Paragraph 1, Acquiring multiple image data from the above image sensor, and A first noise correction map is generated based on the pixel-by-pixel brightness value of each of the plurality of image data, and Check the gain of the first white balance applied to the plurality of image data above, and Calculate the first color temperature using the first white balance gain, and An electronic device that stores the first noise correction map, the gain of the first white balance, and the first color temperature in the memory.

9. In claim 1, the processor, An electronic device that generates an output image by applying the second noise correction map to the image data above, and then deletes the second noise correction map.

10. A method for reducing noise in image data in an electronic device (101 of FIG. 1; 201 of FIG. 2), The operation of acquiring image data from an image sensor of the above electronic device; The operation of obtaining the first noise correction map and the gain of the first white balance from the memory of the electronic device; An operation to generate a second white balance gain based on the above image data; The operation of generating a second noise correction map based on the first noise correction map, the gain of the first white balance, and the second white balance gain; and A method comprising the operation of generating an output image by applying the second noise correction map to the image data.

11. In Paragraph 10, The operation of obtaining a first color temperature representing a color temperature associated with the first noise correction map from the memory; and The operation further includes calculating a second color temperature representing the color temperature of the image data based on the gain of the second white balance, and A method in which the second noise correction map is generated based on a comparison of the first color temperature and the second color temperature.

12. In Paragraph 10, It further includes an operation to determine the reward value, and The above second noise correction map is generated by applying the compensation value to the above first noise correction map, method 13. In Paragraph 12, If the difference between the first color temperature and the second color temperature is greater than or equal to a threshold value, the operation of determining the compensation value based on the gain of the first white balance and the gain of the second white balance; and A method further comprising the operation of determining the compensation value as 1 if the difference between the first color temperature and the second color temperature is less than the threshold value.

14. In Paragraph 10, A method further comprising the operation of applying the second white balance gain to the output image.

15. In a non-volatile storage medium storing instructions, said instructions are configured to cause said electronic device to perform at least one operation when executed by said electronic device, said at least one operation being, The operation of acquiring image data from an image sensor of the electronic device; The operation of obtaining the first noise correction map and the gain of the first white balance from the memory of the electronic device; An operation to generate a second white balance gain based on the above image data; The operation of generating a second noise correction map based on the first noise correction map, the gain of the first white balance, and the second white balance gain; and A storage medium comprising the operation of generating an output image by applying the second noise correction map to the image data.

Images