Method for processing video image, electronic device, and recording medium

Abstract

According to one embodiment, an electronic device comprises: a memory comprising one or more storage media; and at least one processor comprising a processing circuit, wherein instructions, when executed individually or collectively by the at least one processor, may instruct the electronic device to: identify a first image frame; identify a second image frame corresponding to a time point prior to the time point of the first image frame; identify a first region within the second image frame corresponding to a first pixel within the first image frame; identify a correlation between each of a plurality of pixels within the first region and the first pixel within the first image frame; identify a second pixel within the first region on the basis of the correlation; and generate a third image frame on the basis of the second pixel and the first pixel.

Description

Method for processing video images, electronic device, and recording medium

[0001] Embodiments of the present disclosure relate to a method for processing video images, an electronic device, and a recording medium.

[0002] Image quality degradation may occur when shooting with video recording devices, such as digital cameras, in poor conditions like dark environments with relatively low lighting or backlighting. For example, when shooting with a long exposure time set to ensure sufficient exposure, motion blur may occur due to the extended shutter speed, camera shake, or the movement of objects. Additionally, if the camera sensitivity is set to a high level when shooting, dark areas are amplified along with noise components, resulting in strong noise appearing throughout the video.

[0003] To solve the problem of image quality degradation in such low-light environments, technologies such as motion blur removal based on a single resulting image and high-performance noise removal technologies are being developed.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0005] According to one embodiment, the electronic device may include a memory comprising one or more storage media, and at least one processor comprising a processing circuit.

[0006] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to check a first image frame.

[0007] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to identify a second image frame corresponding to a time point earlier than the time point of the first image frame.

[0008] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to identify a first region in the second image frame corresponding to a first pixel in the first image frame.

[0009] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to check the correlation between each of the plurality of pixels in the first region and the first pixel in the first image frame.

[0010] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to identify a second pixel within the first region based on the correlation.

[0011] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may cause to generate a third image frame based on the second pixel and the first pixel.

[0012] According to one embodiment, the method of operating an electronic device may include an operation of checking a first image frame.

[0013] According to one embodiment, the method of operating an electronic device may include the operation of checking a second image frame corresponding to a time point earlier than the time point of the first image frame.

[0014] According to one embodiment, the method of operating an electronic device may include the operation of identifying a first region in the second image frame corresponding to a first pixel in the first image frame.

[0015] According to one embodiment, the method of operating an electronic device may include an operation of checking the correlation between a plurality of pixels within the first region and the first pixel within the first image frame.

[0016] According to one embodiment, the method of operating an electronic device may include the operation of identifying a second pixel within the first region based on the correlation diagram.

[0017] According to one embodiment, the method of operating an electronic device may include the operation of generating a third image frame based on the second pixel and the first pixel.

[0018] According to one embodiment, in a storage medium for storing computer-readable instructions, the instructions cause the electronic device to perform at least one operation when executed by at least one processor of the electronic device, and the at least one operation may include an operation of checking a first image frame.

[0019] According to one embodiment, in a storage medium for storing computer-readable instructions, the instructions cause the electronic device to perform at least one operation when executed by at least one processor of the electronic device, and the at least one operation may include an operation of identifying a second image frame corresponding to a time point earlier than the time point of the first image frame.

[0020] According to one embodiment, in a storage medium for storing computer-readable instructions, the instructions cause the electronic device to perform at least one operation when executed by at least one processor of the electronic device, and the at least one operation may include an operation of identifying a first region in the second image frame corresponding to a first pixel in the first image frame.

[0021] According to one embodiment, in a storage medium for storing computer-readable instructions, the instructions cause the electronic device to perform at least one operation when executed by at least one processor of the electronic device, and the at least one operation may include an operation of checking the correlation between each pixel in the first region and the first pixel in the first image frame.

[0022] According to one embodiment, in a storage medium for storing computer-readable instructions, the instructions cause the electronic device to perform at least one operation when executed by at least one processor of the electronic device, and the at least one operation may include an operation of identifying a second pixel within the first region based on the correlation diagram.

[0023] According to one embodiment, in a storage medium for storing computer-readable instructions, the instructions cause the electronic device to perform at least one operation when executed by at least one processor of the electronic device, and the at least one operation may include an operation of generating a third image frame based on the second pixel and the first pixel.

[0024] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0025] FIG. 2 is a block diagram illustrating a camera module according to one embodiment.

[0026] FIG. 3 is a schematic block diagram of an electronic device according to one embodiment.

[0027] FIG. 4 is a flowchart illustrating the operation of an electronic device processing video images according to one embodiment.

[0028] FIG. 5 is a block diagram showing the structure of an electronic device according to one embodiment.

[0029] FIG. 6 is a block diagram showing the structure of a temporal filter according to one embodiment.

[0030] FIG. 7a is a drawing showing the window area of ​​the current image frame according to one embodiment.

[0031] FIG. 7b is a drawing showing the window area of ​​a previous image frame according to one embodiment.

[0032] FIG. 8 is a block diagram showing the structure of a movement improvement part according to one embodiment.

[0033] FIG. 9 is a block diagram showing the structure of a temporal filter according to one embodiment.

[0034] FIG. 10a is a drawing showing an image to which a motion improvement function is not applied, according to one embodiment.

[0035] FIG. 10b is a drawing showing an image with a motion improvement function applied according to one embodiment.

[0036] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0037] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0038] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence is performed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0039] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0040] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0041] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0042] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0043] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0044] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0045] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0046] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0047] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0048] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0049] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0050] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0051] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0052] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0053] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0054] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally created as part of the antenna module (197).

[0055] According to one embodiment, the antenna module (197) can create a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0056] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0057] According to one embodiment, commands or data may be transmitted or received between an electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0058] FIG. 2 is a block diagram (200) illustrating a camera module (180) according to one embodiment.

[0059] Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is the subject of the image capture. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties different from the lens properties of the other lens assemblies. The lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

[0060] A flash (220) may emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) may include one or more light-emitting diodes (e.g., RGB (red-green-blue) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. An image sensor (230) may acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through a lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) may include, for example, one image sensor selected from image sensors with different properties such as an RGB sensor, a BW (black and white) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same properties, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) may be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0061] The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operational characteristics of the image sensor (230) (e.g., adjusting read-out timing) in response to the movement of the camera module (180) or the electronic device (101) including it. This allows for compensating for at least some of the negative effects caused by the movement on the captured image. According to one embodiment, the image stabilizer (240) can detect such movement of the camera module (180) or the electronic device (101) using a gyroscope sensor (not shown) or an accelerometer sensor (not shown) placed inside or outside the camera module (180). According to one embodiment, the image stabilizer (240) can be implemented, for example, as an optical image stabilizer. The memory (250) can temporarily store at least some of the image acquired through the image sensor (230) for the next image processing operation. For example, when image acquisition by the shutter is delayed or multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in memory (250), and the corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Subsequently, when a specified condition is satisfied (e.g., user input or system command), at least a portion of the original image stored in memory (250) can be acquired and processed, for example, by an image signal processor (260). According to one embodiment, memory (250) may be configured as at least a portion of memory (130) or as a separate memory that operates independently thereof.

[0062] The image signal processor (260) can perform one or more image processing operations on an image obtained through the image sensor (230) or an image stored in memory (250). The above one or more image processing methods may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softing). Additionally or generally, the image signal processor (260) may perform control (e.g., exposure time control, or readout timing control, etc.) over at least one of the components included in the camera module (180) (e.g., image sensor (230)). The image processed by the image signal processor (260) may be stored back in memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) may be composed of at least a part of the processor (120), or It may be configured as a separate processor that operates independently of the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) as is or after additional image processing by the processor (120).

[0063] FIG. 3 is a schematic block diagram of an electronic device according to one embodiment.

[0064] Referring to FIG. 3, according to one embodiment, an electronic device (101) (e.g., the electronic device (101) of FIG. 1) may include a camera module (310) (e.g., the camera module (180) of FIG. 1, the camera module (180) of FIG. 2), a processor (320) (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2), a memory (330) (e.g., the memory (130) of FIG. 1), a communication module (340) (e.g., the communication module (190) of FIG. 1), and a display (360) (e.g., the display module (160) of FIG. 1). According to one embodiment, the electronic device (101) may be implemented identically or similarly to the electronic device (101) of FIG. 1.

[0065] According to one embodiment, the processor (320) can control the overall operation of the electronic device (101). For example, the processor (320) may be implemented as an application processor (AP), a digital signal processor (DSP), or an image signal processor (ISP).

[0066] According to one embodiment, when capturing an image using an electronic device (101) (e.g., in video mode), the processor (320) can acquire an image (e.g., video image) through a camera module (310). The processor (320) can acquire an image (or image data including a plurality of image frames) in units of frames (e.g., image frames) in chronological order. According to one embodiment, the processor (320) can acquire a first image frame and a second image frame. The time point of the second image frame may correspond to a time point earlier than the time point of the first image frame. Hereinafter, for convenience of explanation, the first image frame may be referred to as the current image frame and the second image frame may be referred to as the previous image frame. For example, a second image frame captured at a first time point (t1) may be stored in memory (330), and a first image frame captured at a second time point (t2) after the first time point (t1) may be stored in memory (330). According to various embodiments, the electronic device (101) may acquire an image (or image data including a plurality of image frames) through a communication module (340). The electronic device (101) may transmit the noise-removed image through a processor (320) to an external electronic device (e.g., the electronic device (102, 104) of FIG. 1) or a server (e.g., the server (108) of FIG. 1) through the communication module (340).

[0067] According to one embodiment, noise may be added to an image sensor (e.g., image sensor (230) of FIG. 2) depending on the characteristics of the sensor during the process of acquiring an image (e.g., a plurality of image frames). Noise may be added during the process of transmitting image data acquired through the image sensor through a channel. The added noise may degrade the image quality and reduce the compression performance of the image data. The performance of the degraded image may be degraded during image processing or post-processing through the processor (320).

[0068] According to one embodiment, a method using spatial information and / or a method using temporal information may be applied to remove or reduce the noise. In the various embodiments described below, by utilizing the characteristics of continuously acquired video images, noise occurring at corresponding locations or pixels between temporally different image frames may be removed or reduced by using image information of pixels included in a previous image frame from which noise has been removed.

[0069] According to one embodiment, when noise is removed by utilizing the characteristics of noise occurring at corresponding locations between temporally different image frames, information from previous image frames can be used continuously, so a relatively superior noise reduction effect can be obtained compared to a noise reduction method for still images that uses only spatial information. According to one embodiment, the processor (320) can receive the noise-removed previous image frame as feedback and remove or reduce the noise of the current image frame. According to various embodiments, the pixel or region-specific composite weight of the two image frames can be determined according to the difference between the previous image frame and the current image frame, or according to the difference between a specific region of the previous image frame and a corresponding region of the current image frame. The processor (320) can obtain the current image frame (hereinafter referred to as the third image frame for convenience of explanation) in which noise has been removed or reduced by the composite of the previous image frame and the current image frame.

[0070] According to one embodiment, the processor (320) can identify a first image frame. The processor (320) can identify a second image frame corresponding to a time point earlier than the time point of the first image frame. The processor (320) can identify a first region within the second image frame corresponding to a first pixel within the first image frame. For example, the first region may correspond to an area having a size (e.g., a size of 5 pixels × 5 pixels or 7 pixels × 7 pixels) centered on a pixel corresponding to the position of the first pixel within the second image frame. The first region may be referred to as a search range or a search area, but is not limited to these terms.

[0071] According to one embodiment, the processor (320) can check the correlation between a plurality of pixels within the first area and the first pixel within the first image frame. According to various embodiments, the processor (320) can check the correlation between a second area having a first size (e.g., a size of 3 pixels × 3 pixels) set around the first pixel and a third area (e.g., a sliding window area) having the first size (e.g., a size of 3 pixels × 3 pixels) within the first area. Based on the correlation, the processor (320) can identify a second pixel within the first area (e.g., a search area). According to various embodiments, the second pixel may correspond to the pixel with the highest correlation with the first pixel within the first area. The processor (320) can generate a third image frame based on the second pixel and the first pixel. The third image frame may be an image frame in which noise has been removed or reduced from the first image frame.

[0072] According to one embodiment, the generated third image frame can be stored in memory (330) as an image frame in which noise has been removed or reduced from the first image frame, thereby replacing the first image frame. The third image frame stored in memory (330) can be displayed through a display (360) upon a user's request for playback. Detailed embodiments of removing or reducing noise from the image frame in the processor (320) will be described later.

[0073] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0074] FIG. 4 is a flowchart illustrating the operation of an electronic device processing video images according to one embodiment.

[0075] Referring to FIG. 4, according to one embodiment, in operation 402, an electronic device (101) (e.g., the electronic device (101) of FIG. 3) can check a first image frame.

[0076] According to one embodiment, in operation 404, the electronic device (101) can identify a second image frame corresponding to a time point earlier than the time point of the first image frame.

[0077] According to one embodiment, in operation 406, the electronic device (101) can identify a first region in the second image frame corresponding to a first pixel in the first image frame. For example, the first region may be a region having a size (e.g., a size of 5 pixels × 5 pixels or 7 pixels × 7 pixels) centered on a pixel corresponding to the position of the first pixel in the second image frame. The first region may be referred to as a search range or a search area, but is not limited to these terms.

[0078] According to one embodiment, in operation 408, the electronic device (101) can check the correlation between a plurality of pixels within the first area and the first pixel within the first image frame. According to various embodiments, the electronic device (101) can check the correlation between a second area having a first size (e.g., a size of 3 pixels × 3 pixels) set around the first pixel and a third area (e.g., a slide window area) having the first size (e.g., a size of 3 pixels × 3 pixels) within the first area.

[0079] According to one embodiment, in operation 410, the electronic device (101) can identify a second pixel within the first region (e.g., a search region) based on the correlation. According to various embodiments, the second pixel may correspond to the pixel with the highest correlation with the first pixel within the first region.

[0080] According to one embodiment, in operation 412, the electronic device (101) can generate a third image frame based on the second pixel and the first pixel.

[0081] Hereinafter, with reference to FIGS. 5 to 9, various embodiments for removing or reducing noise in image data in an electronic device (101) will be described in detail.

[0082] FIG. 5 is a block diagram showing the structure of an electronic device according to one embodiment.

[0083] Referring to FIG. 5, an electronic device (101) (e.g., a processor (320) of the electronic device) may include a global motion estimation unit (502), a noise profiling unit (504), a weight map processing unit (506), a noise preprocessing unit (508), a spatial noise processing unit (510), a spatial blending processing unit (512), a temporal blending processing unit (514), and a motion improvement unit (516). At least one of the configurations shown in FIG. 5 may be omitted. Depending on various embodiments, the spatial blending processing unit (512), the temporal blending processing unit (514), and the motion improvement unit (516) may constitute a temporal filter, and a description thereof will be provided in detail later in the description of FIG. 6.

[0084] According to one embodiment, video image data currently being input (e.g., current image frame)) can be input to a global motion estimation unit (502), a noise preprocessing unit (508), a spatial noise processing unit (510), and a spatial blending processing unit (512).

[0085] According to one embodiment, when processing a video image continuously or multiple frames, global movement of the image may occur due to camera movement when comparing the image frame by frame along the time axis. The global movement estimation unit (502) can correct the previous image frame to be mapped to the current image frame by setting the current image frame (e.g., the first image frame) as a reference frame for the previous image frame (e.g., the second image frame, which is an image frame from which noise has been removed or reduced according to a previous noise removal procedure) and warping each pixel. For example, a homography matrix representing the relationship between two image frames can be calculated by using the difference between pixels of the previous image frame and the current image frame (e.g., the difference in brightness values ​​between pixels) or by extracting and matching feature points from each of the two image frames. Using the calculated homography matrix, the current image frame can be set as a reference frame for the previous image frame, and each pixel of the previous image frame can be warped to be corrected to correspond to the current image frame. The global motion estimation unit (502) can transmit the warped and corrected previous image frame to the noise profiling unit (504), weight map processing unit (506), and motion improvement unit (516).

[0086] According to one embodiment, the noise profiling unit (504) receives a warped and corrected previous image frame from the global motion estimation unit (502) and performs noise profiling on the corrected previous image frame to determine the noise level in each region within the previous image frame. The noise profiling unit (504) can transmit the determined noise level in each region to the spatial noise processing unit (510), the noise preprocessing unit (508), the spatial blending processing unit (512), the motion improvement unit (516), and the temporal blending processing unit (514).

[0087] According to one embodiment, the weight map processing unit (506) can generate a weight map (or motion weight map) indicating the degree of local movement by calculating the difference between the warped previous image frame and the current image frame to identify areas with local movement and areas without movement for at least one object included in the image frame. The weight map generated by the weight map processing unit (506) can be transmitted to the spatial noise processing unit (510), the spatial blending processing unit (512), and the temporal blending processing unit (514).

[0088] According to one embodiment, when the weight map processing unit (506) generates the motion weight map, the noise level of the warped previous image frame and the current image frame may not match, so the area with local motion may not be accurately detected. The noise preprocessing unit (508) may process the current image frame such that the noise level of the current image frame is the same or similar to the noise level of the noise-removed previous image frame. For example, the noise preprocessing unit (508) may receive the current image frame and the warped image frame, and transmit the current image frame with the noise level processed to the weight map processing unit (506).

[0089] According to one embodiment, the spatial noise processing unit (510) receives a current image frame and can perform spatial noise removal on the received current image frame based on the noise level of the previous image frame received from the noise profiling unit (504) and the weight map received from the weight map processing unit (506). For example, the spatial noise processing unit (510) can adjust the strength of the filter for spatial noise removal based on the weight map. According to various embodiments, the spatial noise processing unit (510) can set the strength of the filter for spatial noise removal relatively weak in areas where there is relatively little local motion so that detailed image information can be relatively preserved, and in areas where there is relatively large motion, the strength of the filter for spatial noise removal can be set relatively strong so that noise can be removed as much as possible rather than preserving detailed image information for the area, taking into account the image blur caused by the motion. According to various embodiments, a weak filter strength means that the original data is maintained by applying filtering relatively small, and a strong filter strength means that the original data is modified by removing noise from the original data. The spatial noise processing unit (510) can transmit the current image frame with spatial noise removed to the spatial blending processing unit (512). Hereinafter, with reference to FIG. 6, a temporal filter including a spatial blending processing unit (512), a temporal blending processing unit (514), and a motion improvement unit (516) will be described in detail.

[0090] FIG. 6 is a block diagram showing the structure of a temporal filter according to one embodiment.

[0091] Referring to FIG. 6, according to one embodiment, the temporal filter (600) may include a spatial blending processing unit (512), a temporal blending processing unit (514), and a motion improvement unit (516). According to various embodiments, the spatial blending processing unit (512) in the temporal filter (600) may be omitted.

[0092] According to one embodiment, the spatial blending processing unit (512) can add back detailed image information of the current image frame that may have been removed by the spatial noise processing unit (510) by blending the current image frame (605) and the spatial noise removal result (604) based on the weight map (601) received from the weight map processing unit (506).

[0093] According to one embodiment, the time blending processing unit (514) may receive the result of spatial blending (606) for the current image frame from the spatial blending processing unit (512). The time blending processing unit (514) may receive the noise-removed result of the previous image frame transmitted from the global motion estimation unit (502) through the motion improvement unit (516). The time blending processing unit (514) may blend the result of spatial blending (606) for the current image frame with the noise-removed result of the previous image frame based on the motion weight (601) (e.g., local motion weight map) received from the weight map processing unit (506) and the noise level of each region of the previous image frame received from the noise profiling unit (504). For example, the time blending processing unit (514) can maintain image details while maintaining a relatively low noise level in areas with relatively no movement, and output a result with the noise level for the current image frame lowered as much as possible in areas with relatively movement.

[0094] According to one embodiment, if an error in pixel-unit position information occurs in an area where there is no movement between the corrected previous image frame and the current image frame, the detail of the image may be lost by the time blending processing unit (514), and the degree of texture of the flat surface where there is no movement may be degraded. According to various embodiments, the motion improvement unit (516) may find similar pixels within the area around the corresponding pixel of the current image frame and the area around the corresponding pixel of the previous image frame and provide them to the time blending processing unit (514) before performing pixel-by-pixel temporal blending through the time blending processing unit (514) in order to prevent degradation of image quality due to the error in the global motion estimation.

[0095] Hereinafter, with reference to FIGS. 7a, FIGS. 7b and FIGS. 8, a detailed example for finding similar pixels in the motion improvement unit (516) will be described.

[0096] FIG. 7a is a drawing showing the window area of ​​the current image frame according to one embodiment. FIG. 7b is a drawing showing the window area of ​​the previous image frame according to one embodiment.

[0097] Referring to FIGS. 7a and 7b, the pixel with the highest correlation within the previous image frame (720) can be identified for each specific pixel of the current image frame (710). Hereinafter, the current image frame (710) will be referred to as the first image frame, and the previous image frame (720) will be referred to as the second image frame. For example, the motion improvement unit (516) can identify a first region (721) within the second image frame (720) corresponding to a first pixel within the first image frame (710), and can identify the correlation (or similarity) between a plurality of pixels within the first region (721) and the first pixel within the first image frame (e.g., the current image frame (710)). Referring to FIG. 7b, the size of the first region is shown as being set to 7 pixels × 7 pixels, but it is not limited to this size. For example, the size of the first region may be set to 5 pixels × 5 pixels. The motion improvement unit (516) can identify a second pixel within the first region (721) based on the correlation diagram and provide information about the second pixel to the time blending processing unit (514). The time blending processing unit (514) can generate a time-filtered third image frame from the first image frame based on the second pixel and the first pixel.

[0098] According to various embodiments, when checking the correlation between the pixels, an area of ​​a set size (e.g., an area of ​​3 pixels × 3 pixels) can be set as a window, and the correlation between areas of that size can be checked. For example, referring to FIG. 7a, a second area (711) having a first size (e.g., a size of 3 pixels × 3 pixels) set around a first pixel to check the correlation (or similarity) within the first image frame (710) can be set as a window. Referring to FIG. 7b, a first area (721) having a size set around a pixel corresponding to the first pixel of the first image frame (710) can be checked within the second image frame (720). The first area may be referred to as a search range or a search area, but is not limited to these terms. In FIG. 7b above, the search area (722) is shown as having a size of 5 pixels × 5 pixels, but as previously mentioned, when a second area (711) having a first size (e.g., a size of 3 pixels × 3 pixels) centered on the first pixel is set as a window for correlation comparison, the search area or search range can be expanded to a first area (721) having a size of 7 pixels × 7 pixels as shown in FIG. 7b. For example, as illustrated in FIG. 7a and FIG. 7b, when the size of the search area (722) corresponding to the search range for checking the correlation with the first pixel is set to 5 pixels × 5 pixels and the size of the second area (711) corresponding to the window for checking the correlation is set to 3 pixels × 3 pixels, the search area for checking the correlation with the first pixel in the second image frame (720) can be expanded from the search area (722) having a size of 5 pixels × 5 pixels to an area of ​​7 pixels × 7 pixels corresponding to the first area (721).For example, the motion improvement unit (516) can set a window of 3 pixels × 3 pixels, which is the first size of the second area (711), and check the correlation (or similarity) between the second area (711) and the third area (723) corresponding to the size of the window (e.g., 3 pixels × 3 pixels) within the first area (721). The correlation (or similarity) can be checked while moving the third area (723) within the first area (721) corresponding to the search area of ​​the second image frame (720). As a result of checking the correlation (or similarity), the motion improvement unit (516) can set the pixel corresponding to the area with the highest correlation (e.g., the pixel corresponding to the center of the area with the highest correlation) as the second pixel. The time blending processing unit (514) can generate a third image frame with improved temporal noise based on the second pixel that has the highest correlation (or similarity) with the first pixel.

[0099] FIG. 8 is a block diagram showing the structure of a movement improvement part according to one embodiment.

[0100] Referring to FIG. 8, according to one embodiment, the motion improvement unit (516) may include a search window checking unit (810), a current window checking unit (820), a measurement unit (830), and a fallback processing unit (840). The search window checking unit (810) may check a first area (721) or a search area (722) set to check the correlation for a first pixel in a previous image frame (e.g., a second image frame) (720), and a third area (723) for comparison with a second area (711) of the first image frame (710) to check the correlation as a previous window. The current window checking unit (820) may check a second area (721) having a size set around the first pixel as a current window.

[0101] According to one embodiment, the measurement unit (830) can determine the correlation between the current window (e.g., second area (711)) and the previous window (e.g., third area (723)). The measurement unit (830) can measure the correlation between the second area (711) and the third area (723) by applying at least one correlation measurement algorithm using at least one block of SAD (sum of absolute difference) (831), SSD (sum of squared difference) (832), NCC (normalized cross correlation) (833), CT (census transform) (834), and RT (rank transform) (835). According to various embodiments, correlation may be used in a broad sense including similarity. According to one embodiment, the measurement unit (830) may find a pixel with a minimum cost within the search range (e.g., the first area (721)) and transmit the corresponding pixel information and cost (803) to the fallback processing unit (840).

[0102] According to one embodiment, the fallback processing unit (840) can estimate a confidence value that can predict the error rate based on brightness or noise level (602). The fallback processing unit (840) can maintain the original pixel value without applying blending if the pixel with the highest correlation (or similarity) within the search range or search area (e.g., the first area (721) or search area (722)) has a cost higher than a set level. For example, the fallback processing unit (840) can prevent the application of an incorrect pixel value by setting a fallback condition value. According to one embodiment, the fallback processing unit (840) can lower the threshold value of the cost corresponding to the fallback condition relatively because an incorrect motion correction value may be derived when the brightness is dark or the noise level is relatively high, and can relax the fallback condition so that accurate motion correction is applied within the search area if the brightness is relatively bright or the noise level is low.

[0103] FIG. 9 is a block diagram showing the structure of a temporal filter according to one embodiment.

[0104] Referring to FIG. 9, according to one embodiment, the motion improvement unit (516) of FIG. 8 can be expanded into the form of a multilayer temporal filter (900). For example, the multilayer temporal filter (900) can improve the accuracy of the motion improvement unit (516) by configuring the aforementioned motion improvement unit (516) in a pyramid shape.

[0105] According to one embodiment, a pyramid-shaped multilayer temporal filter (900) can correct errors caused by global motion estimation as it progresses from the upper layer to the lower layer. For example, the first motion improvement unit (516a) of the multilayer temporal filter (900) can correct errors caused by global motion estimation for the image frame with the lowest resolution and smallest size. The second motion improvement unit (516b) can receive the processing result of the first motion improvement unit (516a) and correct errors caused by global motion estimation for the image frame with relatively higher resolution and relatively larger size. The third motion improvement unit (516c) can receive the processing result of the second motion improvement unit (516b) and correct errors caused by global motion estimation. The fourth motion improvement unit (516d) can receive the processing result of the third motion improvement unit (516c) and correct errors caused by global motion estimation. The fifth motion improvement unit (516e) can receive the processing result of the fourth motion improvement unit (516d) and correct the error caused by global motion estimation. As described above, it can be applied to relatively large motion estimation errors by hierarchically filtering time in a pyramid shape. Depending on various embodiments, since the noise level and the error of global motion estimation differ for each layer (e.g., for each motion improvement unit), the accuracy of motion improvement can be increased by setting the fallback condition differently for each layer.

[0106] FIG. 10a is a drawing showing an image to which a motion enhancement function is not applied according to one embodiment. FIG. 10b is a drawing showing an image to which a motion enhancement function is applied according to one embodiment. Referring to FIG. 10b, compared to FIG. 10a, it can be seen that the clarity of the image is improved by applying a function that reduces the motion estimation error of the motion enhancement unit (516) as described above.

[0107] According to various embodiments, when removing or reducing temporal noise, noise can be reduced while increasing sharpness and reducing the loss of detail of the current image frame by synthesizing while increasing the alignment rate between the previous image frame and the current image frame. Additionally, according to various embodiments, artifacts caused by motion estimation errors in the global image can be prevented from propagating beyond the current image frame.

[0108] According to one embodiment, the electronic device may include a memory comprising one or more storage media, and at least one processor comprising a processing circuit.

[0109] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be able to identify a first image frame.

[0110] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be able to identify a second image frame corresponding to a time point earlier than the time point of the first image frame.

[0111] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be able to identify a first region in the second image frame corresponding to a first pixel in the first image frame.

[0112] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be able to check the correlation between each of the plurality of pixels in the first region and the first pixel in the first image frame.

[0113] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may identify a second pixel within the first region based on the correlation.

[0114] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may generate a third image frame based on the second pixel and the first pixel.

[0115] According to one embodiment, the first region may have a size set around a pixel corresponding to the position of the first pixel within the second image frame.

[0116] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be able to determine the correlation between a second area having a first size set around the first pixel and a third area having the first size within the first area.

[0117] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device causes the spatial noise of the first image frame to be reduced, and the first pixel may be the pixel with the spatial noise reduced.

[0118] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may generate a weight map related to the movement of at least one object included in the first image frame and reduce spatial noise of the first image frame based on the weight map.

[0119] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be able to adjust the noise level of the first image frame based on information related to the noise level of the second image frame.

[0120] According to one embodiment, the second image frame may be an image frame corrected based on the first image frame.

[0121] According to one embodiment, the second image frame may be an image frame corrected based on warping using a homography matrix.

[0122] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to synthesize the second pixel and the first pixel to generate a third pixel and to generate the third image frame including the third pixel.

[0123] According to one embodiment, the second pixel may correspond to the pixel with the highest correlation within the first region.

[0124] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be able to generate a third image frame based on the first pixel regardless of the second pixel, based on the second pixel having the highest correlation satisfying a set condition.

[0125] According to one embodiment, the method of operating an electronic device may include an operation of checking a first image frame.

[0126] According to one embodiment, the method of operating an electronic device may include the operation of checking a second image frame corresponding to a time point earlier than the time point of the first image frame.

[0127] According to one embodiment, the method of operating an electronic device may include the operation of identifying a first region in the second image frame corresponding to a first pixel in the first image frame.

[0128] According to one embodiment, the method of operating an electronic device may include an operation of checking the correlation between a plurality of pixels within the first region and the first pixel within the first image frame.

[0129] According to one embodiment, the method of operating an electronic device may include the operation of identifying a second pixel within the first region based on the correlation diagram.

[0130] According to one embodiment, the method of operating an electronic device may include the operation of generating a third image frame based on the second pixel and the first pixel.

[0131] According to one embodiment, a method of operating an electronic device, wherein the first region has a size set around a pixel corresponding to the position of the first pixel within the second image frame.

[0132] According to one embodiment, the method may include an operation to check the correlation between a second area having a first size set around the first pixel and a third area having the first size within the first area.

[0133] According to one embodiment, the method further includes an operation to reduce spatial noise of the first image frame, and the first pixel may be a pixel with reduced spatial noise.

[0134] According to one embodiment, the method may further include: generating a weight map related to the movement of at least one object included in the first image frame; and reducing spatial noise of the first image frame based on the weight map.

[0135] According to one embodiment, the method may include an operation of adjusting the noise level of the first image frame based on information related to the noise level of the second image frame.

[0136] According to one embodiment, the second image frame may be an image frame corrected based on the first image frame.

[0137] According to one embodiment, the method may include the operation of synthesizing the second pixel and the first pixel to generate a third pixel; and the operation of generating the third image frame including the third pixel.

[0138] According to one embodiment, in a storage medium for storing computer-readable instructions, the instructions cause the electronic device to perform at least one operation when executed by at least one processor of the electronic device, and the at least one operation may include an operation of checking a first image frame.

[0139] According to one embodiment, in a storage medium for storing computer-readable instructions, the instructions cause the electronic device to perform at least one operation when executed by at least one processor of the electronic device, and the at least one operation may include an operation of identifying a second image frame corresponding to a time point earlier than the time point of the first image frame.

[0140] According to one embodiment, in a storage medium for storing computer-readable instructions, the instructions cause the electronic device to perform at least one operation when executed by at least one processor of the electronic device, and the at least one operation may include an operation of identifying a first region in the second image frame corresponding to a first pixel in the first image frame.

[0141] According to one embodiment, in a storage medium for storing computer-readable instructions, the instructions cause the electronic device to perform at least one operation when executed by at least one processor of the electronic device, and the at least one operation may include an operation of checking the correlation between each pixel in the first region and the first pixel in the first image frame.

[0142] According to one embodiment, in a storage medium for storing computer-readable instructions, the instructions cause the electronic device to perform at least one operation when executed by at least one processor of the electronic device, and the at least one operation may include an operation of identifying a second pixel within the first region based on the correlation diagram.

[0143] According to one embodiment, in a storage medium for storing computer-readable instructions, the instructions cause the electronic device to perform at least one operation when executed by at least one processor of the electronic device, and the at least one operation may include an operation of generating a third image frame based on the second pixel and the first pixel.

[0144] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0145] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, each of phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as “first,” “second,” or “first” or “second” may be used simply to distinguish a component from another component and do not limit the components in any other aspect (e.g., importance or order). Where any (e.g., first) component is referred to as “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicationally,” it means that said component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0146] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0147] Various embodiments of this document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101, 301)). For example, a processor (e.g., processor (120, 320)) of the machine (e.g., electronic device (101, 301)) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' is a device in which the storage medium is tangible, and It merely means that it does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily on a storage medium.

[0148] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0149] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

In the electronic device (101), Memory (130) for storing instructions and including one or more storage media; and It includes at least one processor (120) including a processing circuit, and When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) is enabled, Check the first image frame, Identify a second image frame corresponding to a time point earlier than the time point of the first image frame, and Identifying a first region in the second image frame corresponding to a first pixel in the first image frame, and Checking the correlation between each of the plurality of pixels within the first region and the first pixel within the first image frame, Based on the above correlation diagram, identify the second pixel within the first region, and An electronic device (101) that causes to generate a third image frame based on the second pixel and the first pixel. In paragraph 1, The electronic device (101) wherein the first region has a size set around a pixel corresponding to the position of the first pixel within the second image frame. In paragraph 1 or 2, The above instructions cause the electronic device (101), An electronic device (101) that causes to check the correlation between a second area having a first size set around the first pixel and a third area having the first size within the first area. In any one of paragraphs 1 through 3, The above instructions cause the electronic device (101), Causing to reduce spatial noise of the first image frame, and The above first pixel is an electronic device (101) that is a pixel with reduced spatial noise. In any one of paragraphs 1 through 4, The above instructions cause the electronic device (101), A weight map related to the movement of at least one object included in the first image frame is generated, and An electronic device (101) that causes to reduce spatial noise of the first image frame based on the above weight map. In paragraph 5, The above instructions cause the electronic device (101), An electronic device (101) that adjusts the noise level of the first image frame based on information related to the noise level of the second image frame. In any one of paragraphs 1 through 6, The electronic device (101), wherein the second image frame is an image frame corrected based on the first image frame. In Paragraph 7, The electronic device (101), wherein the second image frame is an image frame corrected based on warping using a homography matrix. In any one of paragraphs 1 through 8, The above instructions cause the electronic device (101), A third pixel is generated by combining the second pixel and the first pixel, and An electronic device (101) that causes the generation of the third image frame including the third pixel. In any one of paragraphs 1 through 9, The electronic device (101) wherein the second pixel corresponds to the pixel with the highest correlation within the first region. In Paragraph 10, The above instructions cause the electronic device (101), An electronic device (101) that generates a third image frame based on the first pixel, regardless of the second pixel, based on the second pixel having the highest correlation satisfying a set condition. In the method of operating the electronic device (101), Action of checking the first image frame; An operation to identify a second image frame corresponding to a point in time earlier than the point in time of the first image frame; An operation to identify a first region within the second image frame corresponding to a first pixel within the first image frame; An operation to check the correlation between a plurality of pixels within the first region and the first pixel within the first image frame; Based on the above correlation diagram, an operation to identify a second pixel within the first region; and A method of operation of an electronic device (101) comprising the operation of generating a third image frame based on the second pixel and the first pixel. In Clause 12, the above method is, The operation of generating a weight map related to the movement of at least one object included in the first image frame; A method of operation of an electronic device (101), further comprising an operation to reduce spatial noise of the first image frame based on the above weight map. In paragraph 12 or 13, the above method is, The operation of synthesizing the second pixel and the first pixel to generate a third pixel; and A method of operation of an electronic device (101) including the operation of generating the third image frame including the third pixel. In a storage medium storing at least one instruction readable by a computer, the at least one instruction causes the electronic device (101) to perform at least one operation when executed by a processor (120) of the electronic device (101). The above at least one operation is: Action of checking the first image frame; An operation to identify a second image frame corresponding to a point in time earlier than the point in time of the first image frame; An operation to identify a first region within the second image frame corresponding to a first pixel within the first image frame; An operation to check the correlation between a plurality of pixels within the first region and the first pixel within the first image frame; Based on the above correlation diagram, an operation to identify a second pixel within the first region; and A storage medium comprising the operation of generating a third image frame based on the second pixel and the first pixel.

Images