Electronic device comprising image processing unit for removing noise from nighttime image and method for operating same

Abstract

The present invention relates to an electronic device comprising an image processing unit that removes noise from a nighttime image, and a method for operating same. The image processing unit may comprise: a noise removal unit that receives a first image, reduces a resolution of the received first image, and then removes noise to generate a second image; and an image restoration unit that receives the second image, calculates information needed to restore the second image, and outputs a third image as a final image by adding the calculated information to the second image.

Description

Electronic device including an image processing unit for removing noise from night images and a method for driving the same

[0001] The present invention relates to an electronic device comprising an image processing unit for removing noise from night images and a method for driving the same.

[0002] At night, due to insufficient light, the camera sensor cannot capture enough light, resulting in dark images and the loss of detailed textures, making it difficult to distinguish subjects. To compensate for this, the sensor sensitivity (ISO value) is increased, but noise inevitably occurs in this process.

[0003] Image noise negatively impacts image quality and usability. Losses caused by noise are particularly pronounced in dark areas or texture representation, while color noise can make images appear patchy in uniform color areas. Furthermore, in image analysis algorithms such as object and face recognition, noise can hinder feature extraction or lead to errors by extracting incorrect information. In addition, noise reduces image compression efficiency, making post-processing tasks (editing, correction, etc.) difficult. Consequently, image noise can cause various problems across overall image utilization, extending beyond mere visual quality degradation. Therefore, effective image noise removal technology is essential to maintain image quality and performance.

[0004] Recently, image noise removal methods utilizing deep learning are being actively researched and developed, and these can be broadly classified into Convolutional Neural Network (CNN)-based methods and Transformer-based methods. CNN-based methods excel at processing local information by learning the relationships between adjacent pixels, making them effective for removing local noise such as Gaussian noise or Salt-and-Pepper noise. On the other hand, Transformer-based methods demonstrate superior performance in removing unstructured and complex noise by considering global information through learning the correlations between pixels or patches across the entire image.

[0005] CNN-based methods enable high-speed processing with relatively few computational resources because they repeatedly perform local convolution operations. However, they have limitations in processing irregular noise spanning wide areas because they only consider the relationships between adjacent pixels. Transformer-based methods complement the weaknesses of CNNs by utilizing global information, demonstrating strengths in removing irregular noise. Nevertheless, they have the disadvantage of exponentially increasing computational costs when processing high-resolution images, as well as high GPU memory requirements and prolonged processing times.

[0006] Various embodiments of the present invention aim to solve the above problems and provide an electronic device and a method for driving the same, comprising an image processing unit based on a new transformer technology suitable for high-resolution image noise removal, which provides a balance between noise removal accuracy and computational efficiency.

[0007] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art from the description below.

[0008] An electronic device according to one embodiment of the present disclosure comprises a control unit, a camera module, a memory, and an image processing unit. The image processing unit includes a noise removal unit that receives a first image, reduces the resolution of the received first image, and then removes noise to generate a second image, and an image restoration unit that receives the second image, calculates information necessary for the restoration of the second image, and adds the calculated information to the second image to output a third image, which is a final image. The noise removal unit includes an image resolution reduction module that reduces the resolution of the first image and increases the number of channels, and a noise removal core module comprising a plurality of transformer modules in a multi-scale hierarchical structure, wherein the plurality of transformer modules include a multi-head attention module that processes global information of the image and a feed-forward module that learns and processes the pattern of the received image by non-linearly transforming the information processed by the multi-head attention module. The noise removal core module also includes an image expansion module that converts an image transmitted from the noise removal core module to the resolution of the first image and removes noise components from the converted image to generate the second image. It can include modules.

[0009] A method for removing noise included in an image using an electronic device according to one embodiment of the present disclosure comprises: a noise removal unit receiving a first image, reducing the resolution of the input first image, and then removing noise to generate a second image; and an image restoration unit receiving the second image, calculating information necessary for the restoration of the second image, and adding the calculated information to the second image to output a third image which is a final image. The noise removal unit may include an image resolution reduction module that reduces the resolution of the first image and increases the number of channels, a noise removal core module comprising a plurality of transformer modules in a multi-scale hierarchical structure, wherein the plurality of transformer modules include a multi-head attention module that processes global information of the image and a feed-forward module that learns and processes the pattern of the input image by non-linearly transforming the information processed by the multi-head attention module, and an image expansion module that converts an image transmitted from the noise removal core module to the resolution of the first image and removes noise components of the converted image to generate the second image.

[0010] According to the embodiments of the present disclosure, noise can be effectively removed from night images while maintaining performance and significantly reducing computational load and GPU memory requirements compared to existing transformer-based methods.

[0011] In addition, various effects that can be identified directly or indirectly through this document may be provided.

[0012] Other aspects, features, and advantages according to specific embodiments of the present disclosure will become more apparent from the accompanying drawings and description.

[0013] FIG. 1 is a block diagram of an electronic device in a network environment according to one embodiment.

[0014] FIG. 2 is a conceptual diagram schematically illustrating a method for removing noise by an image processing unit according to one embodiment.

[0015] FIG. 3 is a block diagram showing the detailed configuration of a noise removal unit according to one embodiment.

[0016] FIG. 4 is a block diagram showing the detailed configuration of a transformer module included in a noise removal core module according to one embodiment.

[0017] FIG. 5 is a block diagram showing the detailed configuration of an image restoration unit according to one embodiment.

[0018] FIG. 6 is a flowchart illustrating a method of driving an electronic device according to one embodiment.

[0019] The aforementioned objectives, features, and advantages are described in detail below with reference to the attached drawings, thereby enabling those skilled in the art to easily implement the technical concept of the present invention. In describing the present invention, detailed descriptions of known technologies related to the present invention are omitted if it is determined that such descriptions would unnecessarily obscure the essence of the invention. Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the attached drawings.

[0020] FIG. 1 is a block diagram showing the components of an electronic device (10) according to one embodiment of the present invention.

[0021] Referring to FIG. 1, an electronic device (10) according to one embodiment may include a control unit (101), a memory (103), a camera module (105), a communication module (107), and an image processing unit (109). The image processing unit (109) may include a noise removal unit (120) and an image restoration unit (140).

[0022] Unlike the illustrated example, the electronic device (10) according to one embodiment may not include at least some of the components illustrated in FIG. 1. Unlike the illustrated example, the electronic device (10) according to one embodiment may further include components not illustrated in FIG. 1 but widely and generally known in the art.

[0023] The electronic device (10) according to the various embodiments disclosed in this disclosure may be of various forms. The electronic device (10) may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device (10) may be a wearable device. A wearable device may include a head-mounted display (HMD) that is wearable on a user's head. A wearable device may be referred to as a head-mount device (HMD), a headgear electronic device, a glasses-type electronic device, a video see-through (VST) or visible see-through (VST) device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device. The electronic device (10) according to the embodiments of this disclosure is not limited to the aforementioned devices.

[0024] According to one embodiment, the control unit (101) may include at least one processor. The processor may, for example, execute software (e.g., a program) to control at least one other component (e.g., a hardware or software component) of the electronic device (10) connected to the processor and perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor may store commands or data received from other components (e.g., a sensor module or a communication module (107)) in volatile memory, process the commands or data stored in volatile memory, and store the resulting data in non-volatile memory. According to one embodiment, the processor may include a main processor (e.g., a central processing unit or an application processor) or an auxiliary processor that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor).

[0025] According to one embodiment, the memory (103) may store various data used by at least one component of the electronic device (10) (e.g., a processor or an image processing unit (109)). The data may include, for example, software (e.g., a program) and input data or output data for related instructions. The memory (103) may include volatile memory or non-volatile memory.

[0026] According to one embodiment, the camera module (105) can capture still images and video. According to one embodiment, the camera module (105) may include one or more lenses, image sensors, image signal processors, or flashes.

[0027] According to one embodiment, the communication module (107) may support the establishment of a wired communication channel or a wireless communication channel between an electronic device (10) and an external electronic device (e.g., a server), and the performance of communication through the established communication channel. According to one embodiment, the communication module (107) may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a GNSS (global navigation satellite system) communication module) or a wired communication module (e.g., a LAN (local area network) communication module, or a power line communication module).

[0028] According to one embodiment, the image processing unit (109) can perform transformer-based noise removal that simultaneously satisfies processing time and noise removal performance. For example, the image processing unit (109) may configure a noise removal block that removes noise in a low-resolution image space with reduced image resolution (size) to shorten processing time, and an image restoration block that restores image loss caused by the reduction in resolution, thereby enabling practical image noise removal that significantly reduces computational complexity and processing time while maintaining existing high performance. The image processing unit (109) may include a noise removal unit (120) and an image restoration unit (140), and their operations will be described in detail later with reference to FIGS. 2 to 5. According to one embodiment, at least some functions of the image processing unit (109) may be performed by the control unit (101). For example, at least some operations performed by the image processing unit (109) may be performed directly by the control unit (101).

[0029] According to one embodiment, the image processing unit (109) may include an artificial intelligence model for image processing. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (10) itself where the artificial intelligence model is performed, or through a separate server (e.g., a server). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above. Artificial intelligence models may include software structures, either additionally or as a substitute, in addition to hardware structures.

[0030] The term “image processing unit (109)” as used in various embodiments of the present disclosure may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as module, logic, logic block, component, or circuit, for example. A module may be a component formed integrally or a minimum unit of said component or a part thereof that performs one or more functions. For example, a module may be implemented in the form of an application-specific integrated circuit (ASIC). The term “module” as used in various embodiments of the present disclosure may be used interchangeably with terms such as “block”.

[0031]

[0032] FIG. 2 is a conceptual diagram schematically illustrating a method for removing noise by an image processing unit (109) according to one embodiment.

[0033] Referring to FIG. 2, an image processing unit (109) according to one embodiment may be composed of two modules (120, 140), and may include, for example, a noise removal unit (120) and an image restoration unit (140).

[0034] According to one embodiment, the noise removal unit (120) removes noise in a low-resolution image space in which the image resolution (size) is reduced to shorten the processing time. For example, the noise removal unit (120) receives a first image (110) as an input image and can reduce the resolution of the input first image (110). After reducing the resolution of the first image (110), the noise removal unit (120) may be configured to remove noise from the reduced image.

[0035] According to one embodiment, the image restoration unit (140) is an image restoration module (or image restoration block) and can perform the role of restoring image quality degradation that occurred during the process of reducing the resolution (e.g., image size) of the image by the noise removal unit (120).

[0036] According to one embodiment, a noise removal unit (120) removes noise from an input image, a first image (110) (e.g., the first image (110)), in a low-resolution image space and outputs a result image (e.g., a second image (130)), and an image restoration unit (140) restores the quality of the result image (e.g., the second image (130)) and outputs a final image (e.g., a third image (150)).

[0037] According to one embodiment, the image noise removal operation performed by the noise removal unit (120) can be expressed as Equation 1.

[0038]

[0039] Referring to Equation 1, Y represents an image containing noise captured by a camera. X represents a clean image without noise. n represents noise information contained in the input image Y. According to Equation 1, if noise information n is removed from the input image Y, a clean image X without noise can be obtained. The image processing unit (109) of the present invention may include a noise removal unit (120) that extracts noise information n from the input image Y based on the principle of Equation 1.

[0040]

[0041] FIG. 3 is a block diagram showing the detailed configuration of a noise removal unit (120) according to one embodiment. For example, the noise removal unit (120) shown in FIG. 3 may be substantially the same as the noise removal unit (120) shown in FIG. 2.

[0042] According to one embodiment, a noise removal unit (120) may receive a first image (210) as an input image and generate a second image (240) by removing noise included in the received first image (210). The noise removal unit (120) may output the generated second image (240). The first image (210) shown in FIG. 3 may be substantially the same as the first image (110) shown in FIG. 2. The second image (240) shown in FIG. 3 may be substantially the same as the second image (130) shown in FIG. 2.

[0043] According to one embodiment, the noise removal unit (120) can calculate noise information (e.g., n of Equation 1) (220) of an input image (e.g., first image (210), Y of Equation 1) (210). According to one embodiment, the noise removal unit (120) can generate a final image with noise removed (e.g., second image (240)) by removing (230) the calculated noise information (e.g., n of Equation 1) (220) from the input image (e.g., first image (210), Y of Equation 1) (210).

[0044] According to one embodiment, the noise removal unit (120) may include an image resolution reduction module (250), a noise removal core module (260), and an image resolution expansion module (270).

[0045] According to one embodiment, the image resolution reduction module (250) can reduce the resolution (size) of the input image (e.g., the first image (210)) and increase the number of channels, and then transmit information (e.g., the first converted image) to the noise removal core module (260).

[0046] According to one embodiment, the noise removal core module (260) may include a plurality of transformer modules (261) in a multi-scale hierarchical structure. According to one embodiment, the result calculated in the noise removal core module (260) (e.g., a second transformed image) may be transmitted to an image resolution expansion module (270). According to one embodiment, the image expansion module (270) may convert the result transmitted from the noise removal core module (260) (e.g., a second transformed image) to the same resolution (size) and number of channels as the input image (e.g., a first image (210)), and remove the noise component (230) to output a final result image (e.g., a second image (240)).

[0047] In the example of FIG. 3, C4S2 included in the noise removal core module (260) represents a convolution layer using a 4x4 filter with a stride of 2, which may be a block that reduces the resolution (size) of the input information by half and increases the number of channels.

[0048] In the example of FIG. 3, CT2S2 included in the noise removal core module (260) represents a transposed convolution layer using a 2x2 filter with a stride of 2, which may be a block for doubling the resolution of input information and reducing the number of channels. In the noise removal core module (260), C represents a Concat operation.

[0049] In the example of FIG. 3, the Pixel Shuffle included in the image resolution expansion module (270) is a block that performs a super-resolution operation to convert a low-resolution image into a high-resolution image, and may be a block that expands the resolution (size) of the result (e.g., second converted image) transmitted to the image expansion module (270) to be the same as the input image (210).

[0050] In the example of FIG. 3, C3S1 included in the image resolution expansion module (270) represents a convolutional layer using a 3x3 filter with a stride of 1, and may be a block that adjusts the number of channels of the result transmitted from the image expansion module (270) and transmits it to Pixel Shuffle.

[0051]

[0052] FIG. 4 is a block diagram showing the detailed configuration of a transformer module (261) included in a noise removal core module (260) according to one embodiment. For example, the transformer module (310) shown in FIG. 4 may be substantially the same as the transformer module (261) shown in FIG. 3.

[0053] Referring to FIG. 4, a transformer module (261, 310) according to one embodiment may include a multi-head attention module (320) and a feed forward module (330).

[0054] According to one embodiment, the multi-head attention module (320) plays a role in effectively processing global information throughout the image and can be designed to learn and process various patterns by processing multiple attention modules (350) in parallel.

[0055] According to one embodiment, the feed-forward module (330) can perform the role of learning and processing more complex patterns by non-linearly transforming the information processed by the multi-head attention module (320).

[0056] In a transformer module (261, 310) according to one embodiment, layer normalization (340) and residual connection (350) can be used to support stable learning in a multi-head attention module (320) and a feed-forward module (330).

[0057] According to one embodiment, the input of the multi-head attention module (320) can be transmitted in the form of an embedding vector.

[0058] According to one embodiment, the multi-head attention module (320) has, as a first step, an input embedding vector I L (341) The query (Q), key (K), and value (V) values ​​can be calculated from. The multi-head attention module (320) applies a weight matrix W to the transmitted embedding vector. q , W k , W v The values ​​of Q (321), K (322), and V (323) can be calculated by multiplying them respectively, and this can be expressed as Equation 2.

[0059] In the example of FIG. 4, the multi-head attention module (320) represents the operation block of multiplying the embedding vector by a weight matrix as Linear (324), and each weight matrix can be optimized during the learning process.

[0060]

[0061]

[0062]

[0063] According to one embodiment, the multi-head attention module (320) can perform an attention (350) process as a second step, which can be expressed mathematically as Equation 3. Referring to Equation 3, the multi-head attention module (320) can calculate the attention weight through a softmax function after calculating the inner product of Q and K during the attention (350) process. The multi-head attention module (320) uses a vector-dimensional square root to prevent the inner product value from becoming excessively large during the attention (350) process. It can be adjusted by dividing it. In the present disclosure, the attention weight represents the relative importance between input elements and may be a factor that determines how much the multi-head attention module (320) should pay attention to each element.

[0064] According to one embodiment, the multi-head attention module (320) can calculate a final result value by multiplying the attention weights by the value (V) as a third step (e.g., the last step).

[0065]

[0066] In the example of Fig. 4, the attention (attention, 350) process is represented in modular units, the inner product of Q and K is represented as MatMul (351), the operation of dividing by the square root of the vector dimension is represented as Normalization (352), the softmax function is represented as Softmax (353), and the operation of multiplying the attention weights by the value is represented as MatMul (354).

[0067] According to one embodiment, as the multi-head attention module (320) processes a plurality of attention modules (350) in parallel, a process of combining each result into one may be required. The multi-head attention module (320) combines a plurality of attention results into one through a Concat (325) operation, and then combines the input embedding vector I through a Linear (326) operation. L The result information can be adjusted with the same size and number of channels as (341).

[0068] According to one embodiment, the feed-forward module (330) can perform the role of learning and processing more complex patterns by non-linearly transforming information that has passed through the multi-head attention module (320). According to one embodiment, the feed-forward module (330) may include at least three convolutional layers (e.g., three convolutional layers) and at least two GELU activation functions (e.g., two GELU activation functions).

[0069] According to one embodiment, in the feedforward module (330), the first convolutional layer, C1S1 (331), uses a 1x1 filter with a stride of 1 to increase the number of channels while maintaining the resolution (size). Subsequently, the feedforward module (330) may add nonlinearity by applying a first GELU activation function (332). For example, GELU can contribute to improving the performance of the model by providing more efficient nonlinear characteristics compared to general activation functions such as RELU or Leaky RELU.

[0070] According to one embodiment, in the feedforward module (330), the second convolutional layer, DC3S1 (333), is a depthwise convolutional layer using a 3x3 filter with a stride of 1, which can process convolutions independently for each channel. The depthwise convolutional layer (i.e., DC3S1 (333)) can reduce memory (103) usage and computational load while maintaining performance. Subsequently, the feedforward module (330) can add nonlinearity by applying a second GELU activation function (334). Finally, the convolutional layer C1S1 (335) of the feedforward module (330) uses a 1x1 filter to reduce the number of channels to be the same as the input of the feedforward module (330).

[0071] As described above, the noise removal unit (120) according to one embodiment removes noise in a low-resolution image space in which the image resolution (size) is reduced (250) to reduce processing time. However, a problem of image quality degradation may occur during this process. The image processing unit (109) according to one embodiment of the present invention may include an image restoration unit (140) to reduce the problem of image quality degradation.

[0072] According to one embodiment, the image restoration unit (140) performs the role of restoring image quality degradation that occurred during the process of reducing the resolution (image size) of the image. The image restoration performed by the image restoration unit (140) according to one embodiment can be defined as Equation 4.

[0073]

[0074] In mathematical equation 4, Y L The output image (240) generated in the noise removal unit (120) is Y H is the reconstructed image, and δ is the input image Y L Each represents the information required for restoration. According to Equation 4, the input image Y L Image Y with improved quality by adding restoration information δ. H ...can be obtained. An image restoration unit (140) according to one embodiment, based on the principle of Equation 4, input image Y L After calculating the restoration information δ for , a calculation can be performed to add it to the input image.

[0075]

[0076] FIG. 5 is a block diagram showing the detailed configuration of an image restoration unit (140) according to one embodiment.

[0077] According to one embodiment, an image restoration unit (140) receives a second image (410) as an input image and can generate a third image (440) by restoring the image quality degradation that occurred during the resolution (image size) reduction process of the received second image (410). The image restoration unit (140) can output the generated third image (440) as a final image. The second image (410) shown in FIG. 5 may be substantially identical to the second image (130) shown in FIG. 2 or the second image (240) shown in FIG. 3. The third image (440) shown in FIG. 5 may be substantially identical to the third image (150) shown in FIG. 2.

[0078] According to one embodiment, the image restoration unit (140) receives a second image (410) as an input image, calculates information δ (420) required for restoration from the input image (e.g., the second image (410)), and then adds the restoration information (420) to the input image (410) (430) to generate a final image (e.g., a third image (440)).

[0079] According to one embodiment, the image restoration unit (140) may be at least partially similar to the noise removal unit (120) described with reference to FIG. 3. According to one embodiment, the image restoration unit (140) may include an image resolution reduction module (450), an image restoration core module (460), and an image resolution expansion module (470).

[0080] According to one embodiment, the image resolution reduction module (450) can reduce the resolution (size) of the input image (410) and increase the number of channels, and then transmit the information to the image restoration core module (460).

[0081] According to one embodiment, the image restoration core module (460) is configured with a multi-scale hierarchical structure and may include a plurality of transformer modules (461). The result calculated in the image restoration core module (460) may be transmitted to the image resolution expansion module (470).

[0082] According to one embodiment, the image expansion module (470) can convert the result transmitted from the image restoration core module (460) to the same resolution and number of channels as the input image (e.g., second image (410)), and then sum (430) it with the input image to output a final image (e.g., third image (440)).

[0083] In the example of FIG. 5, C4S2 represents a convolutional layer using a 4x4 filter with a stride of 2, and the convolutional layer C4S2 can reduce the resolution (size) of the input information by half and increase the number of channels. CT2S2 is an inverse convolutional layer using a 2x2 filter with a stride of 2, which can expand the resolution of the input information by a factor of 2 and reduce the number of channels. Pixel Shuffle can expand the resolution of the result transmitted from the image expansion module (470) to be equal to the input image (410). C3S1 is a convolutional layer using a 3x3 filter with a stride of 1, which can perform the role of adjusting the number of channels of the result transmitted from the image expansion module (470) and transmitting it to Pixel Shuffle.

[0084] The detailed structure of the transformer module (461) included in the image restoration core module (460) may have substantially the same structure as the transformer module (261) included in the noise removal core module (260). Therefore, the description regarding the operation of the transformer module (461) included in the image restoration core module (460) will be replaced by the description regarding the transformer module (261) included in the noise removal core module (260).

[0085] Various embodiments of the present invention can provide a transformer-based image noise removal method that maintains performance while reducing computational complexity and processing time. Here, the computational complexity of the transformer's multi-head attention can be expressed as Equation 5.

[0086]

[0087] In mathematical formula 5, h represents the vertical length of the input information, w represents the horizontal length, and c represents the number of channels. According to mathematical formula 5, the computational complexity of the transformer increases in proportion to the square of the product of the horizontal length and the vertical length of the image. Therefore, if the image size is reduced by half, the amount of computation can be reduced by approximately 1 / 16. The problem caused by reducing the image size by half was solved by adding an image restoration unit (140), and by using two modules, the amount of computation of the noise removal method proposed in the present invention was reduced by approximately 1 / 8 compared to the existing transformer-based method, while showing similar performance results.

[0088]

[0089] FIG. 6 is a flowchart illustrating a method of driving an electronic device according to one embodiment.

[0090] The operations illustrated in FIG. 6 can be performed by instructions stored in memory (e.g., memory (103) of FIG. 1). For example, when the instructions are executed by a control unit (e.g., control unit (101) of FIG. 1), an electronic device (e.g., electronic device (10) of FIG. 1) can perform the operations illustrated in FIG. 6.

[0091] In operation 610, the image processing unit (109) can receive the first image (110). For example, the image processing unit (109) can receive the first image (110) captured through the camera module (105). For example, the image processing unit (109) can receive the first image (110) stored in the memory (103).

[0092] In operation 620, the noise removal unit (120) of the image processing unit (109) removes noise in a low-resolution image space in which the image resolution (size) has been reduced in order to shorten the processing time. For example, the noise removal unit (120) may receive a first image (110) as an input image and reduce the resolution of the input first image (110). After reducing the resolution of the first image (110), the noise removal unit (120) may be configured to remove noise from the reduced image.

[0093] According to one embodiment, the noise removal unit (120) may include an image resolution reduction module (250), a noise removal core module (260), and an image resolution expansion module (270).

[0094] According to one embodiment, the image resolution reduction module (250) can reduce the resolution (size) of the input image (e.g., the first image (110)) and increase the number of channels, and then transmit information (e.g., the first converted image) to the noise removal core module (260).

[0095] According to one embodiment, the noise removal core module (260) may include a plurality of transformer modules (261) in a multi-scale hierarchical structure. According to one embodiment, the result calculated in the noise removal core module (260) (e.g., a second transformed image) may be transmitted to an image resolution expansion module (270). According to one embodiment, the image expansion module (270) may convert the result transmitted from the noise removal core module (260) (e.g., a second transformed image) to the same resolution (size) and number of channels as the input image (e.g., a first image (110)), and remove the noise component (230) to output a final result image (e.g., a second image (130)).

[0096] In operation 630, the image restoration unit (140) can perform the role of restoring image quality degradation that occurred during the process of reducing the resolution (e.g., image size) of the image by the noise removal unit (120).

[0097] According to one embodiment, the image restoration unit (140) may include an image resolution reduction module (450), an image restoration core module (460), and an image resolution expansion module (470).

[0098] According to one embodiment, the image resolution reduction module (450) can reduce the resolution (size) of the input image (410) and increase the number of channels, and then transmit the information to the image restoration core module (460).

[0099] According to one embodiment, the image restoration core module (460) is configured with a multi-scale hierarchical structure and may include a plurality of transformer modules (461). The result calculated in the image restoration core module (460) may be transmitted to the image resolution expansion module (470).

[0100] According to one embodiment, the image expansion module (470) can convert the result transmitted from the image restoration core module (460) to the same resolution and number of channels as the input image (e.g., second image (130)), and then sum (430) it with the input image to output a final image (e.g., third image (440)).

[0101] Compared to existing transformer-based methods, this invention can significantly reduce computational load and GPU memory requirements while maintaining performance. Therefore, it is expected to be applicable not only to camera image correction applications but also to various fields such as intelligent CCTV or autonomous driving, where GPU capacity is relatively limited and object or face recognition can be applied.

[0102] Although the present invention has been illustrated and described above in relation to a specific preferred embodiment, it is obvious to those skilled in the art that the present invention can be modified and varied in various ways without departing from the technical features or scope of the invention as defined by the following claims.

Claims

1. In an electronic device, Control unit; Camera module; Memory; and It includes an image processing unit, and the image processing unit, A noise removal unit that receives a first image, reduces the resolution of the input first image, and then removes noise to generate a second image; and It includes an image restoration unit that receives the second image, calculates information necessary for the restoration of the second image, and outputs a third image, which is a final image, by adding the calculated information to the second image. The above noise removal unit is, An image resolution reduction module that reduces the resolution of the first image and increases the number of channels; A noise removal core module comprising a plurality of transformer modules in a multi-scale hierarchical structure, wherein the plurality of transformer modules include a multi-head attention module that processes global information of an image and a feed-forward module that learns and processes patterns of an input image by non-linearly transforming the information processed by the multi-head attention module; and An image expansion module comprising converting an image transmitted from the noise removal core module to the resolution of the first image and removing noise components from the converted image to generate the second image, Electronic device.

2. In Paragraph 1, The above multi-head attention module is As a first step, a weight matrix W is applied to the input embedding vector I. q , W k , W v Calculate the query (Q), key (K), and value (V) values ​​by multiplying them respectively, and In the second step, the attention process, the inner product of the above query and the above key is calculated, and then the attention weight is calculated through the softmax function, and In the third step, set to perform a calculation of multiplying the attention weight by the value, Electronic device.

3. In Paragraph 2, The above multi-head attention module is The square root of the vector dimension in the above attention process Set to perform calculations by , Electronic device.

4. In Paragraph 3, The above multi-head attention module combines multiple attention results into one through a Concat operation, and then combines the above input embedding vector I through a Linear operation. L Set to adjust result information with the same size and number of channels as, Electronic device.

5. In Paragraph 1, The above feedforward module comprises at least three convolutional layers and at least two GELU activation functions, Electronic device.

6. In Paragraph 1, The above image restoration unit An image resolution reduction module that reduces the resolution of an input image and increases the number of channels; An image restoration core module composed of a multi-scale hierarchical structure and including a plurality of transformer modules; and The image expansion module comprises converting the result transmitted from the image restoration core module to the same resolution and number of channels as the second image, and then adding the converted image with the input image to generate the third image. Electronic device.

7. A method for an electronic device to remove noise included in an image, A noise removal unit receives a first image, reduces the resolution of the input first image, and then removes noise to generate a second image; and The image restoration unit includes the operation of receiving the second image, calculating information necessary for the restoration of the second image, and adding the calculated information to the second image to output a third image, which is the final image. The above noise removal unit is, An image resolution reduction module that reduces the resolution of the first image and increases the number of channels; A noise removal core module comprising a plurality of transformer modules in a multi-scale hierarchical structure, wherein the plurality of transformer modules include a multi-head attention module that processes global information of an image and a feed-forward module that learns and processes patterns of an input image by non-linearly transforming the information processed by the multi-head attention module; and An image expansion module comprising converting an image transmitted from the noise removal core module to the resolution of the first image and removing noise components from the converted image to generate the second image, method.

8. In Paragraph 7, The above multi-head attention module is As a first step, a weight matrix W is applied to the input embedding vector I. q , W k , W v Calculate the query (Q), key (K), and value (V) values ​​by multiplying them respectively, and In the second step, the attention process, the inner product of the above query and the above key is calculated, and then the attention weight is calculated through the softmax function, and In the third step, set to perform a calculation of multiplying the attention weight by the value, method.

9. In Paragraph 8, The above multi-head attention module is The square root of the vector dimension in the above attention process Set to perform calculations by , method.

10. In Paragraph 9, The above multi-head attention module combines multiple attention results into one through a Concat operation, and then combines the above input embedding vector I through a Linear operation. L Set to adjust result information with the same size and number of channels as, method.

Images