Denoising method and device using neural model
A neural model architecture with a neural control, denoising, and text recognition model iteratively enhances image quality and text recognition, addressing the limitations of existing models in low-quality conditions.
Patent Information
- Application Number
- PCT/KR2026/005759
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-08-28
- Filing Date
- 2026-04-30
- Publication Date
- 2026-11-12
AI Technical Summary
Existing neural network models struggle to effectively denoise and recognize text in images, particularly in low-quality conditions, due to limitations in generalization and accuracy for untrained patterns.
A neural model architecture comprising a neural control model, a neural denoising model, and a neural text recognition model, trained through a diffusion and reverse process, is employed to generate guide features and denoised images, improving image quality and text recognition accuracy over time steps.
The model enhances image quality by removing noise and improves text recognition accuracy by iteratively refining denoised images and text recognition results, achieving high-quality super-resolution images with precise text information and location detection.
Smart Images

Figure KR2026005759_12112026_PF_FP_ABST
Abstract
Description
Denoising method and device using a neural model
[0001] The following embodiments relate to a denoising method and apparatus using a neural model.
[0002] Neural network models implemented using processors to possess specialized computational structures can generate computationally intuitive mappings between input and output patterns after substantial training. This trained ability to generate such mappings can be referred to as the learning capability of the neural network model. A neural network model trained through specialized training can possess generalization capabilities, for example, by generating relatively accurate outputs for untrained input patterns. Neural network models can be utilized for image restoration.
[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.
[0004] According to one embodiment, the electronic device may include one or more processors comprising processing circuits. According to one embodiment, the electronic device may include a memory for storing instructions. When the instructions are executed individually or collectively by one or more processors, they may cause the electronic device to execute a neural control model based on an input image to generate guide features. When the instructions are executed individually or collectively by one or more processors, they may cause the electronic device to execute a neural denoising model based on a previous denoised image, a previous text recognition result, and guide features to generate a current denoised image. When the instructions are executed individually or collectively by one or more processors, they may cause the electronic device to execute a neural text recognition model based on current denoising features generated by one or more layers of the neural denoising model to generate a current text recognition result to generate a current denoised image.
[0005] According to one embodiment, the electronic device may include one or more processors comprising processing circuits. According to one embodiment, the electronic device may include a memory for storing instructions. When the instructions are executed individually or collectively by one or more processors, the electronic device may cause a neural control model to execute based on the current time step, an input image, and a previous text recognition result to generate guide features. When the instructions are executed individually or collectively by one or more processors, the electronic device may cause a neural denoising model to execute based on the current time step, a previous denoised image, a previous text recognition result, and guide features to generate a current denoised image. When the instructions are executed individually or collectively by one or more processors, the electronic device may cause a detection decoder of a neural text recognition model to execute based on current denoising features generated by one or more layers of a neural denoising model to generate location information of a current text recognition result to generate a current denoised image. When instructions are executed individually or collectively by one or more processors, they can cause an electronic device to execute a recognition decoder of a neural text recognition model based on current denoising features to generate text information of the current text recognition result.
[0006] According to one embodiment, the processor-implemented method may include an operation of generating guide features by executing a neural control model based on an input image. The processor-implemented method may include an operation of generating a current denoised image by executing a neural denoising model based on a previous denoised image, a previous text recognition result, and the guide features. The processor-implemented method may include an operation of generating a current text recognition result by executing a neural text recognition model based on current denoising features generated by one or more layers of the neural denoising model to generate the current denoised image.
[0007] FIG. 1 is a drawing illustrating the configuration of an electronic device according to one embodiment.
[0008] FIG. 2 is a block diagram exemplarily illustrating the configuration of a camera module according to one embodiment.
[0009] FIG. 3 is a diagram illustrating, in an exemplary manner, the configuration of a neural model that performs image generation according to one embodiment.
[0010] FIG. 4 is a diagram exemplarily illustrating the structure of a neural control model and a neural denoising model according to one embodiment.
[0011] FIG. 5 is a diagram exemplarily illustrating the structure of a neural text recognition model according to one embodiment.
[0012] FIG. 6 is a diagram illustrating another configuration of a neural model that performs image generation according to one embodiment.
[0013] FIG. 7 is a flowchart exemplarily illustrating an image generation operation using a neural model according to one embodiment.
[0014] FIG. 8 is a diagram exemplarily illustrating a diffusion process and a reverse process used for training a neural model according to one embodiment.
[0015] FIG. 9 is a diagram exemplarily showing a training loss used for training a neural model according to one embodiment.
[0016] FIG. 10 is a flowchart exemplarily illustrating the operation of training a neural model according to one embodiment.
[0017] FIG. 11 is a flowchart exemplarily illustrating a denoising method using a neural model according to one embodiment.
[0018] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are given the same reference numeral regardless of the drawing number, and redundant descriptions thereof will be omitted.
[0019] FIG. 1 is a diagram illustrating the configuration of an electronic device according to one embodiment. Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through the server (108).
[0020] According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).
[0021] The processor (120) may be implemented as one or more IC (integrated circuit (or circuitry)) chips and may perform various data processing operations. The processor (120) may include at least one electrical circuit and may process instructions (or programs (140), data, etc.) stored in memory (130) individually or collectively in a distributed manner. The processor (120) may include a processor assembly comprising one or more processing circuits. The processor (120) may include any processing circuit that is operative to control the performance and operation of one or more components of the electronic device (101) (e.g., memory (130), display module (160), camera module (180), communication module (190), and / or sensor (176)).
[0022] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.
[0023] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.
[0024] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related instructions. The memory (130) may include volatile memory (132) or non-volatile memory (134).
[0025] According to one embodiment, the electronic device (101) may include one or more processors (120) including processing circuits. The electronic device (101) may include a memory (130) for storing instructions. When the instructions are executed individually or collectively by one or more processors (120), the electronic device (101) may be caused to execute a neural control model based on an input image to generate guide features. When the instructions are executed individually or collectively by one or more processors (120), the electronic device (101) may be caused to execute a neural denoising model based on a previous denoised image, a previous text recognition result, and guide features to generate a current denoised image. To generate a current denoised image, the electronic device (101) may be caused to execute a neural text recognition model based on current denoising features generated by one or more layers of the neural denoising model to generate a current text recognition result.
[0026] According to one embodiment, a previously denoised image may be generated using a neural denoising model at a time step prior to the generation of the current denoised image. A previously text recognition result may be generated using a neural text recognition model at a time step prior to the generation of the current text recognition result.
[0027] According to one embodiment, one or more layers of a neural denoising model can generate current denoising features based on a previously denoised image, a previous text recognition result, and guide features. A neural text recognition model can generate a current text recognition result based on the current denoising features.
[0028] According to one embodiment, the neural denoising model may include an encoding block, a middle block, and a decoding block. When instructions are executed individually or collectively by one or more processors (120), the electronic device (101) may further cause to generate an output of a first layer based on the output of a previous layer of a first layer of the decoding block, the output of a corresponding layer of a first layer of the encoding block, and the output of a corresponding layer of a first layer of the neural control model, and to execute a neural text recognition model based on current denoising features including the output of the first layer.
[0029] According to one embodiment, the neural denoising model may include an encoding block, a middle block, and a decoding block. One or more layers may belong to at least a part of the middle block and the decoding block.
[0030] According to one embodiment, the current text recognition result may include one or more of text information regarding text included in the current denoised image and location information regarding the location of the text. The neural text recognition model may generate text information and location information as the current text recognition result. When the instructions are executed individually or collectively by one or more processors (120), the electronic device (101) may further cause the neural denoising model to selectively input text information among text information and location information as the previous text recognition result into the neural denoising model when executing the neural denoising model.
[0031] According to one embodiment, when instructions are executed individually or collectively by one or more processors (120), the electronic device (101) may further cause the previous text recognition result to be fused into the output of each layer of the neural denoising model.
[0032] According to one embodiment, when instructions are executed individually or collectively by one or more processors (120), the electronic device (101) may further cause to input a previous time step into a neural denoising model when generating a previous denoised image and input a current time step into a neural denoising model when generating a current denoised image.
[0033] According to one embodiment, when instructions are executed individually or collectively by one or more processors (120), the electronic device (101) may further cause the neural denoising model and the neural text recognition model to be executed repeatedly according to time steps. The text recognition results of the neural text recognition model may be updated as the neural denoising model and the neural text recognition model are executed repeatedly.
[0034] According to one embodiment, when instructions are executed individually or collectively by one or more processors (120), the electronic device (101) may further cause to execute a neural denoising model without a text recognition result of a neural text recognition model during a preset initial interval of time steps.
[0035] According to one embodiment, when instructions are executed individually or collectively by one or more processors (120), the electronic device (101) may further cause to classify the quality level of an input image into a first quality level or a second quality level lower than the first quality level, and if the quality level is classified into the first quality level, to execute a neural denoising model using text recognition results during a preset initial period, and if the quality level is classified into the second quality level, to execute a neural denoising model without text recognition results during a preset initial period.
[0036] According to one embodiment, the electronic device (101) may include one or more processors (120) including processing circuits. The electronic device (101) may include a memory (130) for storing instructions. When the instructions are executed individually or collectively by one or more processors (120), they may cause the electronic device (101) to execute a neural control model based on the current time step, an input image, and a previous text recognition result to generate guide features. When the instructions are executed individually or collectively by one or more processors (120), they may cause the electronic device (101) to execute a neural denoising model based on the current time step, a previous denoised image, a previous text recognition result, and guide features to generate a current denoised image. When the instructions are executed individually or collectively by one or more processors (120), the electronic device (101) may be caused to execute a detection decoder of a neural text recognition model based on current denoising features generated by one or more layers of a neural denoising model to generate a currently denoised image, thereby generating location information of the current text recognition result. When the instructions are executed individually or collectively by one or more processors (120), the electronic device (101) may be caused to execute a recognition decoder of a neural text recognition model based on current denoising features, thereby generating text information of the current text recognition result.
[0037] According to one embodiment, one or more layers of a neural denoising model can generate current denoising features based on a previously denoised image, a previous text recognition result, and guide features. A neural text recognition model can generate a current text recognition result based on the current denoising features.
[0038] According to one embodiment, the neural denoising model includes an encoding block, a middle block, and a decoding block, and one or more layers may belong to at least a part of the middle block and the decoding block. The neural text recognition model can generate text information and location information as a result of current text recognition. When the instructions are executed individually or collectively by one or more processors (120), the electronic device (101) may further cause the neural denoising model to selectively input text information among text information and location information as a result of previous text recognition into the neural denoising model.
[0039] According to one embodiment, when instructions are executed individually or collectively by one or more processors (120), the electronic device (101) may further cause the current time step and previous text recognition results to be fused into the output of each layer of the neural control model and the output of each layer of the neural denoising model.
[0040] According to one embodiment, when instructions are executed individually or collectively by one or more processors (120), the electronic device (101) may further cause the neural denoising model and the neural text recognition model to be executed repeatedly according to time steps. The text recognition results of the neural text recognition model may be updated as the neural denoising model and the neural text recognition model are executed repeatedly.
[0041] According to one embodiment, when instructions are executed individually or collectively by one or more processors (120), the electronic device (101) may further cause to execute a neural denoising model without a text recognition result of a neural text recognition model during a preset initial interval of time steps.
[0042] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).
[0043] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).
[0044] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.
[0045] A display module (160) (e.g., a display) can visually provide information to an external (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.
[0046] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).
[0047] The sensor (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor. For example, the sensor (176) may include an inertial measurement unit (IMU).
[0048] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.
[0049] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).
[0050] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.
[0051] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, one or more image sensors, one or more image signal processors, or one or more flashes.
[0052] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).
[0053] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.
[0054] A communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication circuits. The communication module (190) may include one or more communication processors (CP) that operate independently of a processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local region network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct or IrDA (infrared data relation)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips).The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network, such as a first network (198) or a second network (199), using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).
[0055] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.
[0056] An antenna module (197) can transmit a signal or power to an external source (e.g., an external electronic device) or receive it from an external source. According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).
[0057] According to one embodiment, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.
[0058] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.
[0059] According to one embodiment, commands or data may be transmitted or received between an electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199).
[0060] Each of the external electronic devices (102, 104) and the server (108) may be of the same or different type as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104) or the server (108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the request may perform at least part of the requested function or service, or additional functions or services related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request.
[0061] FIG. 2 is a block diagram illustrating the configuration of a camera module according to one embodiment. Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260) (e.g., the processor (120) of FIG. 1, an auxiliary processor (123)). The lens assembly (210) may collect light emitted from a subject that is the target of image capture. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties different from the lens properties of other lens assemblies. The lens assemblies (210) may include, for example, a wide-angle lens or a telephoto lens.
[0062] A flash (220) may emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) may include one or more light-emitting diodes (e.g., RGB (red-green-blue) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. An image sensor (230) may acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through a lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) may include, for example, one image sensor selected from image sensors with different properties such as an RGB sensor, a BW (black and white) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same properties, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.
[0063] The image stabilizer (240) may move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction in response to the movement of the camera module (180) or the electronic device (101) including it, or control the operational characteristics of the image sensor (230) (e.g., adjusting read-out timing). This allows for compensating for at least some of the negative effects caused by the movement on the image being captured. According to one embodiment, the image stabilizer (240) may detect such movement of the camera module (180) or the electronic device (101) using a gyroscope sensor (not shown) or an accelerometer sensor (not shown) placed inside or outside the camera module (180).
[0064] According to one embodiment, the image stabilizer (240) may be implemented, for example, as an optical image stabilizer. The memory (250) may temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing operation. For example, when image acquisition by the shutter is delayed or multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and the corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Subsequently, when a specified condition is satisfied (e.g., user input or system command), at least a portion of the original image stored in the memory (250) may be acquired and processed, for example, by an image signal processor (260). According to one embodiment, the memory (250) may be configured as at least a portion of the memory (130) or as a separate memory that operates independently thereof.
[0065] The image signal processor (260) can perform one or more image processing operations on an image obtained through the image sensor (230) or an image stored in memory (250). The above one or more image processing methods may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softing). Additionally or generally, the image signal processor (260) may perform control (e.g., exposure time control, or readout timing control, etc.) over at least one of the components included in the camera module (180) (e.g., image sensor (230)). The image processed by the image signal processor (260) may be stored back in memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) is at least part of the processor (120). It may be configured as a separate processor that operates independently of the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) as is or after additional image processing by the processor (120).
[0066] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180) each having different attributes or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera and at least another may be a rear camera.
[0067] FIG. 3 is a diagram illustrating, in an exemplary manner, the configuration of a neural model used for image generation according to one embodiment. Referring to FIG. 3, the neural model (300) may include a neural control model (310), a neural denoising model (320), and a neural text recognition model (330). The neural control model (310), the neural denoising model (320), and the neural text recognition model (330) may each include a deep learning-based neural network.
[0068] Image restoration may correspond to an image processing technique that enhances the quality of an input image (301). For example, the input image (301) may correspond to a low-quality image caused by low-quality factors such as noise, blur, compression loss, and resolution degradation. An electronic device (e.g., the electronic device (101) of FIG. 1) may perform image restoration using a neural model (300) to remove or mitigate low-quality factors from the input image (301) and generate a high-quality image (e.g., a high-quality super-resolution image) corresponding to the input image (301).
[0069] The neural model (300) can be pre-trained based on a diffusion process and a reverse process. For example, in the diffusion process, Gaussian noise can be added stepwise to the original image to generate noisy images, and in the reverse process, Gaussian noise can be removed stepwise from the noisy images to restore the original image. The operations for training the neural model (300) will be described in more detail later.
[0070] The electronic device can perform image restoration on an input image (301) by repeatedly executing a neural model (300) according to a time step. The operation of executing a model such as the neural model (300) may include operations that process operations constituting the model. As a result of image restoration, an output image (e.g., a high-quality super-resolution image) may be generated. At each iteration, the electronic device may update the current time step (302). If the current time step (302) is the last time step, the current denoised image (305) may correspond to the output image.
[0071] More specifically, the electronic device may execute a neural control model (310) based on an input image (301) to generate a guide feature (308). For example, the neural control model (310) may include a neural encoder and perform an encoding operation to extract the guide feature (308) from the input image (301), but is not limited thereto. The neural control model (310) may guide the denoising direction of the neural denoising model (320) based on the guide feature (308).
[0072] The electronic device can generate a current denoised image (305) by running a neural denoising model (320) based on a previous denoised image (303), a previous text recognition result (304), and guide features (308). The previous denoised image (303) can be generated using the neural denoising model (320) at a time step prior to the generation of the current denoised image (305), and the previous text recognition result (304) can be generated using the neural text recognition model (330) at a time step prior to the generation of the current text recognition result. For example, a previous denoised image (303) and a previous text recognition result (304) may be generated at a previous time step (not shown), and a current denoised image (305) and a current text recognition result (307) may be generated at a current time step (302).
[0073] According to one embodiment, the neural denoising model (320) may be pre-trained to predict the current denoised image (305) or pre-trained to predict the previous noise of the previous denoised image (303). In the former case, the electronic device may execute the neural denoising model (320) to obtain the current denoised image (305) from the output of the neural denoising model (320). In the latter case, the electronic device may execute the neural denoising model (320) to obtain the previous noise of the previous denoised image (303) from the output of the neural denoising model (320), and obtain the current denoised image (305) by subtracting the previous noise from the previous denoised image (303).
[0074] When the electronic device executes the neural denoising model (320), it may input a time step (e.g., current time step (302)) into the neural denoising model (320). The electronic device may update the time step over time. For example, when the electronic device generates a previously denoised image (303) at a previous time step (not shown), it may input a previous time step (not shown) into the neural denoising model (320), and when generating a currently denoised image (305) at the current time step (302), it may input a current time step (302) into the neural denoising model (320). For example, the current time step (302) may be represented by t. t may be a natural number from T to 1. T may represent the total number of time steps. In the diffusion process, t can be updated from 1 to T, and in the reverse process, t can be updated from T to 1. FIG. 3 may represent a neural model (300) that has completed training, and the inference operation of the neural model (300) that has completed training may correspond to the reverse process.
[0075] The electronic device may first execute a neural denoising model (320) based on the current time step (302) of T, and then execute a neural denoising model (320) based on the current time step (302) of T-1. The previously denoised image (303) at the current time step (302) of T may be a pure noise image. Finally, the electronic device may execute a neural denoising model (320) based on the current time step (302) of 1. The current denoised image (305) generated by executing the neural denoising model (320) based on the current time step (302) of 1 may correspond to an output image (e.g., a high-quality super-resolution image).
[0076] The neural denoising model (320) may include one or more layers (e.g., layer (320a)). One or more layers of the neural denoising model (320) may generate a current denoising feature (306) to generate a currently denoised image (305). The electronic device may run a neural text recognition model (330) based on the current denoising feature (306) to generate a current text recognition result (307). For example, the neural text recognition model (330) may perform neural network-based optical character recognition (OCR), but is not limited thereto.
[0077] The text recognition result (e.g., previous text recognition result (304), current text recognition result (307)) may include one or more of text information regarding text contained in the denoised image (e.g., previous denoised image (303), current denoised image (305)) and location information regarding the location of the text. For example, the text information may include information regarding one or more texts (e.g., characters). For example, the location information may include information (e.g., coordinates) regarding a polygon (e.g., a box) representing the location of the text.
[0078] According to one embodiment, when the electronic device executes the neural denoising model (320), text information and location information may be optionally input to the neural denoising model (320). For example, the electronic device may optionally input text information among the text information and location information to the neural denoising model (320). The neural denoising model (320) may be trained to generate text information and location information as a result of text recognition. The electronic device may optionally input text information among the text information and location information generated by the neural denoising model (320) to the neural denoising model (320).
[0079] The electronic device may repeatedly execute at least a portion of the neural model (300) according to time steps. For example, the electronic device may input the current denoised image (305) and the current text recognition result (307) into the neural denoising model (320) at the next time step. The text recognition result of the neural text recognition model (330) may be updated as the neural denoising model (320) and the neural text recognition model (330) are repeatedly executed. For example, the current text recognition result (307) may differ from the previous text recognition result (304) depending on the difference between the previous denoised image (303) and the current denoised image (305). As the neural denoising model (320) and the neural text recognition model (330) are executed repeatedly, the quality of the denoised image (e.g., previous denoised image (303), current denoised image (305)) can be gradually improved, and as a result, the recognition accuracy of the text recognition result can also be improved.
[0080] According to one embodiment, the electronic device may execute the neural denoising model (320) without the text recognition result of the neural text recognition model (330) during a preset initial interval among the time steps. For example, the electronic device may execute the neural denoising model (320) based on the current time step (302) and the previous denoised image (303). For example, the electronic device may input a predetermined value into the neural text recognition model (330) instead of the text recognition result, but is not limited thereto. In some cases, an inaccurate text recognition result may be generated during the initial interval. By excluding the text recognition result during the initial interval, it may be prevented that the text recognition result interferes with the denoising operation of the neural denoising model (320).
[0081] According to one embodiment, the electronic device may determine whether to exclude text recognition results during an initial period based on the quality level of the input image (301). For example, the electronic device may exclude text recognition results during an initial period if the quality level of the input image (301) is lower than a predetermined quality threshold. The electronic device may utilize text recognition results during an initial period if the quality of the input image (301) is higher than the quality threshold. For example, the quality levels may include a first quality level (e.g., high quality) higher than the threshold and a second quality level (e.g., low quality) lower than the threshold. The quality threshold may be a quality level value between the first quality level (e.g., high quality) and the second quality level (e.g., low quality). The second quality level may be lower than the first quality level. When the quality level of the input image (301) is determined to be a first quality level, the electronic device can execute a neural denoising model (320) using text recognition results during an initial period, and when the quality level is determined to be a second quality level, it can execute a neural denoising model (320) without text recognition results during an initial period.
[0082] According to one embodiment, the electronic device may classify the quality of an input image (301) into a plurality of quality levels based on a quality factor. For example, the quality factor may include, but is not limited to, one or more of the shooting settings, shooting environment, and visual components of the input image (301). The shooting settings are the setting values of the camera that took the input image (301), and may include, for example, resolution, ISO sensitivity, shutter speed, aperture value, and white balance, but is not limited to. The shooting environment is the environment in which the input image (301) was taken, and may include illuminance, lighting type, humidity, and fog, but is not limited to. For example, the electronic device may measure the shooting environment using a sensor (e.g., sensor (176) of FIG. 1). The visual components are components that appear visually in the input image (301), and may include, but are not limited to, resolution, sharpness, noise, contrast, brightness, and distortion.
[0083] According to one embodiment, the initial interval used to exclude text recognition results when the quality of the input image (301) is low may be a time step interval in which the quality of the previous denoised image (303) and / or the current denoised image (305) does not exceed a predetermined quality threshold. The current time step (302) may be represented as t. t may be a natural number from T to 1. For example, as the time steps progress from T, the quality level of the previous denoised image (303) and / or the current denoised image (305) may gradually improve due to the execution of the neural denoising model (320), and the electronic device may execute the neural denoising model (320) without text recognition results until the quality level of the previous denoised image (303) and / or the current denoised image (305) exceeds the quality threshold. For example, time steps can proceed from T according to a reverse process (e.g., the reverse process (820) of FIG. 8). As time steps progress, from when the previous denoised image (303) and / or the current denoised image (305) exceeds a quality threshold, the electronic device can execute a neural denoising model (320) using the text recognition results.
[0084] According to one embodiment, the input image (301) may include image components and text components. The image components may include visual information with low-frequency characteristics and visual information with high-frequency characteristics. For example, edges appearing in the image may have high-frequency characteristics, and planes appearing in the image may have low-frequency characteristics. Text components may generally have high-frequency characteristics. When a general diffusion model is used to restore an input image (301) in which image components and text components are mixed, the text portion may not be clearly restored. When a diffusion model trained exclusively for text restoration is used, it may be required to crop the text portion and input it into the diffusion model.
[0085] A neural model (300) according to one embodiment may include a neural text recognition model (330) that generates a text recognition result (e.g., current text recognition result (307)) using a denoising feature (e.g., current denoising feature (306)). According to one embodiment, the text recognition result may be used as an input to a neural denoising model (320). According to one embodiment, due to the structure and / or operation of such a neural model (300), both the image component and the text component of the input image (301) may be restored in a balanced manner.
[0086] FIG. 4 is a diagram illustrating the structure of a neural control model and a neural denoising model according to one embodiment. Referring to FIG. 4, a neural control model (410) (e.g., the neural control model (310) of FIG. 3) may include an encoding block (411) and a middle block (412). The middle block (412) may correspond to a bottleneck. The encoding block (411) and the middle block (412) may each include one or more layers (e.g., layer (410a)).
[0087] A neural denoising model (420) (e.g., neural denoising model (320)) may include an encoding block (421), a middle block (422), and a decoding block (423). The middle block (422) may correspond to a bottleneck. The encoding block (421), the middle block (422), and the decoding block (423) may each include one or more layers (e.g., layer (420a), layer (420b)).
[0088] According to one embodiment, the encoding block (411) may have the same structure as the encoding block (421), but is not limited thereto. According to one embodiment, the middle block (412) may have the same structure as the middle block (422), but is not limited thereto. According to one embodiment, the input of the neural control model (410) and the input of the neural denoising model (420) may correspond to each other, but are not limited thereto. In this case, the electronic device may generate guide features (408) by executing the neural control model (410) based on the current time step (402), the input image (401), and the previous text recognition result (404).
[0089] One or more layers of the neural control model (410) can generate guide features (408) based on the input image (401). For example, one or more layers of the neural control model (410) can generate guide features (408) based on the current time step (402), the input image (401), and the previous text recognition result (404). One or more layers of the neural denoising model (420) can generate the current denoised image (405) based on the current time step (402), the previous denoised image (403), the previous text recognition result (404), and the guide features (408).
[0090] One or more layers of the neural denoising model (420) can generate a current denoising feature (406) based on a current time step (402), a previous denoising image (403), a previous text recognition result (404), and a guide feature (408) to generate a current denoising image (405). According to one embodiment, one or more layers generating the current denoising feature (406) may belong to at least a part of a middle block (422) and a decoding block (423). A neural text recognition model (e.g., the neural text recognition model (330) of FIG. 3) can generate a current text recognition result based on the current denoising feature (406).
[0091] Layers of the encoding block (411), middle block (412), encoding block (421), middle block (422), and decoding block (423) may have a corresponding relationship with each other based on input size and / or output size. Layers of the encoding block (411), middle block (412), encoding block (421), middle block (422), and decoding block (423) that have the same input size and / or the same output size may be called corresponding layers. For example, layer (410a) of the encoding block (411), layer (420b) of the encoding block (421), and layer (420a) of the decoding block (423) may have a corresponding relationship with each other.
[0092] Each layer (e.g., layer (420a)) of the middle block (422) and the decoding block (423) may operate based on the output of the corresponding layer. For example, one or more layers of the neural denoising model (420) may include a first layer (e.g., layer (420a)) of the decoding block (423). The electronic device may generate the output of the first layer based on the output of the previous layer of the first layer of the decoding block (423), the output of the corresponding layer (e.g., layer (420b)) of the first layer of the encoding block (421), and the output of the corresponding layer (e.g., layer (410a)) of the first layer of the neural control model (410) (e.g., encoding block (411)). For example, the electronic device may generate a computation result of the first layer based on the output of the previous layer of the first layer, and may generate an output of the first layer by fusing the computation result of the first layer, the output of the corresponding layer of the first layer of the encoding block (421), and the output of the corresponding layer of the first layer of the neural control model (410) (e.g., layer (410a)). For example, the fusion may include, but is not limited to, summation and / or concatenation. The neural denoising model (420) may include, but is not limited to, skip connections between the corresponding layers of the encoding block (421) and the decoding block (423). The output of the corresponding layer of the first layer of the encoding block (421) (e.g., layer (420b)) may be part of the encoded features (409) transmitted through the skip connections.
[0093] According to one embodiment, the electronic device may fuse a time step (e.g., current time step (402)) and a text recognition result (e.g., previous text recognition result (404)) with the output of each layer of the neural denoising model (420). For example, the fusion may include, but is not limited to, summing and / or chaining. If the neural control model (410) has the same structure as the neural denoising model (420), the electronic device may fuse the time step and the text recognition result with the output of each layer of the neural control model (410).
[0094] FIG. 5 is a diagram illustrating the structure of a neural text recognition model according to one embodiment. Referring to FIG. 5, the neural text recognition model (500) may include a feature encoder (520), a detection decoder (530), and a recognition decoder (540). The feature encoder (520) and the detection decoder (530) may correspond to a neural encoder and a neural decoder. An electronic device (e.g., the electronic device (101) of FIG. 1) may execute the neural text recognition model (500) based on current denoising features (501) (e.g., the current denoising features (306) of FIG. 3, the current denoising features (406) of FIG. 4) to generate a current text recognition result (505).
[0095] According to one embodiment, the electronic device can generate processed features (502) by processing the current denoising features (501) in the feature processing block (510). For example, the electronic device can generate processed features (502) by flattening the current denoising features (501) in the feature processing block (510). The electronic device can execute a neural text recognition model (500) based on the processed features (502).
[0096] The electronic device can perform size transformation of the current denoising feature (501) using a convolution layer (511) in the feature processing block (510). For example, layers of the decoding block (e.g., decoding block (423)) of the neural denoising model (e.g., neural denoising model (320) of FIG. 3, neural denoising model (420) of FIG. 4) can generate output features of different sizes, and the current denoising feature (501) may include output features of these different sizes. The electronic device can convert output features of different sizes into intermediate features of the same size using the convolution layer (511). For example, the intermediate features may be one-dimensional features. The electronic device can generate processed features (502) by performing a connection (512) of the intermediate features. For example, the connection (512) may include chaining.
[0097] The electronic device can execute a feature encoder (520) based on processed features (502). For example, the electronic device can tokenize the processed features (502) to generate input tokens, and execute a feature encoder (520) based on the input tokens to generate output tokens. The input tokens may include sub-tokens for a specific region (e.g., a patch), and the output tokens may each correspond to the processing results of the sub-tokens.
[0098] The layers of the feature encoder (520) (e.g., layer (521)) may include a transformer layer. For example, the transformer layer may include one or more attention layers (e.g., self-attention layer) and one or more multilayer perceptron (MLP) layers. The attention layer may perform an attention mechanism.
[0099] According to one embodiment, the electronic device may select some of the output tokens as target tokens based on the regions and classes of the output tokens. For example, the number of output tokens may be N, and the number of target tokens may be K. The electronic device may generate features corresponding to the output tokens and predict class scores for each feature. The electronic device may select target tokens from the output tokens in order of highest text class score, which indicates the probability that the feature corresponds to text.
[0100] The electronic device can generate a current text recognition result (505) by executing a detection decoder (530) and a recognition decoder (540) based on target tokens. Layers of the detection decoder (530) (e.g., layer (531)) and layers of the recognition decoder (540) (e.g., layer (541)) may include a transformer layer. For example, the transformer layer may include one or more attention layers (e.g., cross-attention layer) and one or more MLP layers. The attention layer may perform an attention mechanism.
[0101] The current text recognition result (505) may include location information (503) and text information (504). The electronic device may generate location information (503) by executing a detection FFN group (feed-forward network group) (550) based on the output of a detection decoder (530). The electronic device may generate text information (504) by executing a recognition FFN group (560) based on the output of a recognition decoder (540). For example, the location information (503) may include information (e.g., coordinates) regarding a polygon (e.g., a box) representing the location of the text. The text information (504) may include information regarding one or more texts (e.g., characters).
[0102] According to one embodiment, each of the detection FFN group (550) and the recognition FFN group (560) may include K FFNs. Each of the K FFNs of the detection FFN group (550) can process the corresponding target token of the K target tokens to generate sub-information of the corresponding target token among the location information (503). Each of the K FFNs of the recognition FFN group (560) can process the corresponding target token of the K target tokens to generate sub-information of the corresponding target token among the text information (504).
[0103] FIG. 6 is a diagram illustrating another configuration of a neural model used for image generation according to one embodiment. Referring to FIG. 6, the neural model (600) (e.g., the neural model (300) of FIG. 3) may include a neural control model (610) (e.g., the neural control model (310) of FIG. 3, the neural control model (410) of FIG. 4) and a neural denoising model (620) (e.g., the neural denoising model (320) of FIG. 3, the neural denoising model (420) of FIG. 4).
[0104] The neural control model (610) can generate guide features (608) based on the input image (601). The neural denoising model (620) can generate a current denoised image (605) based on the current time step (602) and the previous denoised image (603). The current denoised image (605) can be input to the neural denoising model (620) at the next time step.
[0105] The neural model (600) of FIG. 6 may not include a neural text recognition model (e.g., neural text recognition model (330)) compared to the neural model (300) of FIG. 3. For example, the neural model (600) may be trained with a neural text recognition model included, and after training is completed, may perform an inference operation without a neural text recognition model, but is not limited thereto.
[0106] In this case, the neural model (600) may have the ability to restore the image and text components of the input image (601) in a balanced manner due to the neural text recognition model during the training process. Additionally, the size of the neural model (600) may be reduced as the neural text recognition model is excluded from the neural model (600) during inference. As a result, the neural model (600) may be operated with less memory and / or fewer operations in environments where computing resources and / or memory resources are limited, such as mobile or embedded environments.
[0107] FIG. 7 is a flowchart illustrating an image generation operation using a neural model according to one embodiment. Referring to FIG. 7, an electronic device (e.g., the electronic device (101) of FIG. 1) can generate guide features (708) by executing a neural control model (e.g., the neural control model (310) of FIG. 3, the neural control model (410) of FIG. 4, and the neural control model (610) of FIG. 6) based on an input image (701) in operation (710).
[0108] The electronic device may generate a denoised image (705) (e.g., current denoised image) by executing a neural denoising model (e.g., neural denoising model (320) of FIG. 3, neural denoising model (420) of FIG. 4, neural denoising model (620) of FIG. 6) based on a time step (702) (e.g., current time step), a denoised image (705) (e.g., previous denoised image), a text recognition result (707) (e.g., previous text recognition result) and a guide feature (708) in operation (720). For example, if the time step (702) is the first time step, the denoised image (705) may be a pure noise image (e.g., Gaussian noise image), and / or the text recognition result (707) may be a null value, but is not limited thereto. One or more layers of the neural denoising model can generate denoising features (706) (e.g., current denoising features) to generate a denoised image (705) (e.g., current denoised image).
[0109] The electronic device can check in operation (730) whether the current time step is the last time step. If the current time step is the last time step, the electronic device can output the last denoised image in operation (750). The last denoised image may correspond to the output image (751).
[0110] If the current time step is not the last time step, the electronic device may generate a text recognition result (707) (e.g., current text recognition result) by executing a neural text recognition model (e.g., neural text recognition model (330) of FIG. 3, neural text recognition model (500) of FIG. 5) based on a denoising feature (706) (e.g., current denoising feature) in operation (740). The electronic device may perform operation (720) again in the next time step. For example, the electronic device may generate a denoised image (705) (e.g., next denoised image) by executing a neural denoising model based on a time step (702) (e.g., next time step), a denoised image (705) (e.g., current denoised image), a text recognition result (707) (e.g., current text recognition result), and a guide feature (708).
[0111] FIG. 8 is a diagram exemplarily illustrating a diffusion process and a reverse process used for training a neural model according to one embodiment. Referring to FIG. 8, images (x0 to x) are obtained through a diffusion process (810). T-1 Noise can be gradually added to ). The addition of noise can be called noising. Through a diffusion process (810), an image (x) which is a pure noise image is obtained from an original image (image (x0). T ) can be generated. For example, at time step t, image(x t-1 Noise is added to the image(x t ) can be generated. t can be from 1 to T. In the diffusion process (810), t can increase from 1 to T. For example, initially, noise can be added to image (x0) to generate image (x1), and finally, image (x T-1 Noise is added to the image(x T) can be generated. The diffusion process (810) can be performed based on a predetermined noise function (e.g., a Gaussian noise function).
[0112] Gradual noise removal can be achieved through a reverse process (820). Noise removal can be called denoising. For example, at time step t, the image (x t Noise is removed from the image (y t-1 )(not shown) can be generated. In the reverse process (820), t can decrease from T to 1. In the next type step t-1, the image (y t-1 Noise is removed from the image (y t-2 )(Unvisited) can be generated. Initially, an image(x T Noise is removed from the image (y T-1 )(not shown) can be generated, and finally, noise can be removed from image (y1) to generate image (y)(not shown). Image (y) may correspond to an output image.
[0113] Training data to be used for training a neural model (e.g., neural model (300) of FIG. 3, neural model (600) of FIG. 6) in the diffusion process (810) can be determined. For example, training data for the reverse process (820) can be determined based on a noise image and / or added noise. For example, when training a neural denoising model (e.g., neural denoising model (320) of FIG. 3, neural denoising model (420) of FIG. 4, neural denoising model (620) of FIG. 6) to predict a denoised image, training data can be determined based on the noise image. When training a neural denoising model to predict noise, training data can be determined based on the added noise. In the reverse process (820), the neural model can be trained based on the training data.
[0114] FIG. 9 is a diagram exemplarily illustrating a training loss used for training a neural model according to one embodiment. Referring to FIG. 9, a neural model (900) (e.g., neural model (300) of FIG. 3, neural model (600) of FIG. 6) may include a neural control model (910) (e.g., neural control model (310) of FIG. 3, neural control model (410) of FIG. 4, neural control model (610) of FIG. 6), a neural denoising model (920) (e.g., neural denoising model (320) of FIG. 3, neural denoising model (420) of FIG. 4, neural denoising model (620) of FIG. 6), and a neural text recognition model (930) (e.g., neural text recognition model (330) of FIG. 3, neural text recognition model (500) of FIG. 5). An electronic device (e.g., electronic device (101) of FIG. 1, server (108)) can run a neural model (900) based on an input image (901) to determine a training loss (e.g., denoise loss (L_dno), text loss (L_txt), position loss (L_pos)) and train the neural model (900) based on the training loss. The electronic device may also be called a training device.
[0115] More specifically, the electronic device may generate guide features (908) by running a neural control model (910) based on an input image (901). The electronic device may generate a current denoised image (905) by running a neural denoising model (920) based on the current time step (902), a previous denoised image (903), a previous text recognition result (904), and guide features (908). The previous denoised image (903) and the previous text recognition result (904) may be generated at a previous time step. The electronic device may generate a current text recognition result (907) by running a neural text recognition model (930) based on the current denoising features (906). The current denoising features (906) may be generated by one or more layers of the neural denoising model (920) to generate the current denoised image (905). The current text recognition result (907) may include text information (9071) and location information (9072).
[0116] The electronic device can iteratively run the neural control model (910), the neural denoising model (920), and the neural text recognition model (930) while increasing the current time step (902). When the current time step (902) is the last time step, the electronic device can determine the training loss (e.g., denoising loss (L_dno) corresponding to the difference between the current denoised image (905) and the label image (953), text loss (L_txt) corresponding to the difference between text information (9071) and the label text (951), and position loss (L_pos) corresponding to the difference between position information (9072) and the label position (952).
[0117] For example, the input image (901) may be a low-resolution version of the original image (e.g., image (x0) of FIG. 8). The label image (953) may be the original image. The label text (951) may correspond to a text component included in the label image (953). The label location (952) may indicate the location of the text component included in the label image (953). The label image (953) may refer to a ground truth (GT) image. In this case, the label text (951) and the label location (952) may refer to a GT text component and a location of the GT text component, respectively, included in the GT image.
[0118] The electronic device can update the neural model (900) based on training loss. For example, the electronic device can update the parameters (e.g., weights) of the neural control model (910), the neural denoising model (920), and the neural text recognition model (930) so that the training loss is reduced. The electronic device can repeatedly train the neural model (900) based on a predetermined number of time steps.
[0119] According to one embodiment, the electronic device may determine a training loss based on a weighted sum of denoise loss (L_dno), text loss (L_txt), and position loss (L_pos). The weights may be determined based on user preferences. For example, user preferences may include, but are not limited to, an image-priority setting that prioritizes the restoration of image components and a text-priority setting that prioritizes the restoration of text components.
[0120] When an image-first setting is used, a higher weight may be assigned to the denoise loss (L_dno) compared to the text loss (L_txt) and position loss (L_pos). When a text-first setting is used, a higher weight may be assigned to the text loss (L_txt) and position loss (L_pos) compared to the denoise loss (L_dno). The electronic device may train a first sub-neural model of the neural model (900) based on an image-first setting and train a second sub-neural model of the neural model (900) based on a text-first setting, but is not limited thereto. The first sub-neural model and the second neural model may each include corresponding models of a neural control model (910), a neural denoising model (920), and a neural text recognition model (930).
[0121] The first subneural model and the second subneural model may be mounted on an electronic device (e.g., the electronic device (101) of FIG. 1). The user may selectively use the first subneural model and the second subneural model according to user preference. For example, an image restoration mode may include an image-first restoration mode and a text-first restoration mode. If the image-first restoration mode is selected by the user, the first subneural model may be used for image restoration. If the text-first restoration mode is selected, the second subneural model may be used for image restoration.
[0122] FIG. 10 is a flowchart illustrating, in an exemplary manner, the operation of training a neural model according to one embodiment. Referring to FIG. 10, an electronic device (e.g., the electronic device (101) of FIG. 1, a server (108)) can generate guide features (1008) by executing a neural control model (e.g., the neural control model (310) of FIG. 3, the neural control model (410) of FIG. 4, the neural control model (610) of FIG. 6, the neural control model (910) of FIG. 9) based on an input image (1001) in an operation (1010).
[0123] The electronic device may generate a denoised image (1005) (e.g., current denoised image) by executing a neural denoising model (e.g., neural denoising model (320) of FIG. 3, neural denoising model (420) of FIG. 4, neural denoising model (620) of FIG. 6, neural denoising model (920) of FIG. 9) based on a time step (1002) (e.g., current time step), a denoised image (1005) (e.g., previous denoised image), and a text recognition result (1007) (e.g., previous text recognition result) in operation (1020). If the time step (1002) is the first time step, the denoised image (1005) may be a pure noise image (e.g., Gaussian noise image), and / or the text recognition result (1007) may be a null value, but is not limited thereto. One or more layers of the neural denoising model can generate denoising features (1006) (e.g., current denoising features) to generate a denoised image (1005) (e.g., current denoised image).
[0124] The electronic device can check in operation (1030) whether the current time step is the last time step. If the current time step is not the last time step, the electronic device can in operation (1040) execute a neural text recognition model (e.g., neural text recognition model (330) of FIG. 3, neural text recognition model (500) of FIG. 5, neural text recognition model (930) of FIG. 9) based on a denoising feature (1006) (e.g., current denoising feature) to generate a text recognition result (1007) (e.g., current text recognition result). If the current time step is the last time step, the electronic device can in operation (1050) calculate a training loss (e.g., denoising loss (L_dno), text loss (L_txt), position loss (L_pos) of FIG. 9). The electronic device can in operation (1060) perform a model update based on the training loss.
[0125] FIG. 11 is a flowchart illustrating an exemplary denoising method using a neural model according to one embodiment. Referring to FIG. 11, in operation (1110), an electronic device (e.g., the electronic device (101) of FIG. 1, a server (108)) can generate guide features by executing a neural control model (e.g., the neural control model (310) of FIG. 3, the neural control model (410) of FIG. 4, the neural control model (610) of FIG. 6, the neural control model (910) of FIG. 9) based on an input image. In operation (1120), the electronic device can generate a current denoised image by running a neural denoising model (e.g., the neural denoising model (320) of FIG. 3, the neural denoising model (420) of FIG. 4, the neural denoising model (620) of FIG. 6, and the neural denoising model (920) of FIG. 9) based on a previous denoised image, a previous text recognition result, and guide features. In operation (1130), the electronic device can generate a current text recognition result by running a neural text recognition model (e.g., the neural text recognition model (330) of FIG. 3, the neural text recognition model (500) of FIG. 5, and the neural text recognition model (930) of FIG. 9)) based on current denoising features generated by one or more layers of the neural denoising model to generate a current denoised image.
[0126] According to one embodiment, a previously denoised image may be generated using a neural denoising model at a time step prior to the generation of the current denoised image. A previously text recognition result may be generated using a neural text recognition model at a time step prior to the generation of the current text recognition result.
[0127] According to one embodiment, one or more layers of a neural denoising model can generate current denoising features based on a previously denoised image, a previous text recognition result, and guide features. A neural text recognition model can generate a current text recognition result based on the current denoising features.
[0128] According to one embodiment, the neural denoising model may include an encoding block, a middle block, and a decoding block. The electronic device may generate an output of the first layer based on the output of the previous layer of the first layer of the decoding block, the output of the corresponding layer of the first layer of the encoding block, and the output of the corresponding layer of the first layer of the neural control model, and may execute a neural text recognition model based on current denoising features including the output of the first layer.
[0129] According to one embodiment, the neural denoising model may include an encoding block, a middle block, and a decoding block. One or more layers may belong to at least a part of the middle block and the decoding block.
[0130] According to one embodiment, the current text recognition result may include one or more of text information regarding text contained in the currently denoised image and location information regarding the location of the text. The neural text recognition model may generate text information and location information as the current text recognition result. When the electronic device executes the neural denoising model, it may further cause the neural denoising model to selectively input text information among the text information and location information as a previous text recognition result.
[0131] According to one embodiment, the electronic device may further cause the previous text recognition results to be fused into the output of each layer of the neural denoising model.
[0132] According to one embodiment, the electronic device may input a previous time step into a neural denoising model when generating a previously denoised image, and input a current time step into a neural denoising model when generating a currently denoised image.
[0133] According to one embodiment, the electronic device can iteratively execute a neural denoising model and a neural text recognition model according to time steps. The text recognition result of the neural text recognition model can be updated as the neural denoising model and the neural text recognition model are iteratively executed.
[0134] According to one embodiment, the electronic device can execute a neural denoising model without the text recognition result of the neural text recognition model during a preset initial interval among the time steps.
[0135] According to one embodiment, the electronic device classifies the quality level of an input image into a first quality level or a second quality level lower than the first quality level, and when the quality level is classified into the first quality level, it executes a neural denoising model using text recognition results during a preset initial period, and when the quality level is classified into the second quality level, it executes a neural denoising model without text recognition results during a preset initial period.
[0136] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains.
[0137] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs.
[0138] An electronic device according to one embodiment disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone, a tablet computer), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. An electronic device according to an embodiment of this document is not limited to the aforementioned devices.
[0139] One embodiment of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.
[0140] The term "module" as used in an embodiment of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to an embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).
[0141] One embodiment of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.
[0142] According to one embodiment, the method according to one embodiment disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.
[0143] According to one embodiment, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to one embodiment, one or more of the components or operations among the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to one embodiment, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.
[0144] The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.
[0145] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, or computer storage medium or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on computer-readable recording media.
[0146] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination, and the program instructions recorded on the medium may be those specifically designed and configured for the embodiment or those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.
[0147] The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.
Claims
1. In an electronic device (101; 108), One or more processors (120) including processing circuits; and Memory (130) for storing instructions Includes, When the above instructions are executed individually or collectively by the one or more processors, the electronic device, Run a neural control model based on the input image to generate guide features, and A neural denoising model is executed based on the previous denoised image, the previous text recognition result, and the guide features above to generate the current denoised image, and A neural text recognition model is executed based on current denoising features generated by one or more layers of the neural denoising model to generate the above-mentioned currently denoised image, thereby causing the generation of a current text recognition result. Electronic device.
2. In Paragraph 1, The previously denoised image above The above-mentioned currently denoised image is generated using the above-mentioned neural denoising model at a time step prior to the generation of the above-mentioned currently denoised image, and The above previous text recognition result is A neural text recognition model generated using the neural text recognition model at a time step prior to the generation of the current text recognition result. Electronic device.
3. In Paragraph 1 or 2, The one or more layers of the above neural denoising model are Generate the current denoising feature based on the above-mentioned previously denoised image, the above-mentioned previously text recognition result, and the above-mentioned guide feature, and The above neural text recognition model Generating the current text recognition result based on the above current denoising feature, Electronic device.
4. In any one of paragraphs 1 through 3, The above neural denoising model It includes an encoding block, a middle block, and a decoding block, When the above instructions are executed individually or collectively by the one or more processors, the electronic device, Generating the output of the first layer based on the output of the previous layer of the first layer of the decoding block, the output of the corresponding layer of the first layer of the encoding block, and the output of the corresponding layer of the first layer of the neural control model. Further causing to execute the neural text recognition model based on the current denoising feature including the output of the first layer, Electronic device.
5. In any one of paragraphs 1 through 4, The above neural denoising model It includes an encoding block, a middle block, and a decoding block, The above one or more layers are at least part of the above middle block and the above decoding block, Electronic device.
6. In any one of paragraphs 1 through 5, The above current text recognition result is A text information regarding text included in the above-mentioned currently denoised image and a location information regarding the location of said text, comprising one or more of said text information. Electronic device.
7. In Paragraph 6, The above neural text recognition model Generate the text information and location information based on the current text recognition result, and When the above instructions are executed individually or collectively by the one or more processors, the electronic device, When executing the above neural denoising model, further causing the above neural denoising model to selectively input the text information among the text information and the location information as the previous text recognition result, Electronic device.
8. In any one of paragraphs 1 through 7, When the above instructions are executed individually or collectively by the one or more processors, the electronic device, Further causing the aforementioned previous text recognition results to be fused into the output of each layer of the neural denoising model, Electronic device.
9. In any one of paragraphs 1 through 8, When the above instructions are executed individually or collectively by the one or more processors, the electronic device, When generating the aforementioned previously denoised image, the previous time step is input into the neural denoising model, and When generating the currently denoised image, causing the current time step to be input into the neural denoising model, Electronic device.
10. In any one of paragraphs 1 through 9, When the above instructions are executed individually or collectively by the one or more processors, the electronic device, Further causing the above neural denoising model and the above neural text recognition model to be executed repeatedly according to time steps, The text recognition result of the above neural text recognition model is updated as the above neural denoising model and the above neural text recognition model are executed iteratively, Electronic device.
11. In Paragraph 10, When the above instructions are executed individually or collectively by the one or more processors, the electronic device, Further causing the neural denoising model to be executed without the text recognition result of the neural text recognition model during a preset initial interval among the above time steps, Electronic device.
12. In Paragraph 11, When the above instructions are executed individually or collectively by the one or more processors, the electronic device, The quality level of the above input image is classified into a first quality level or a second quality level lower than the first quality level, and If the above quality level is classified as the above first quality level, the neural denoising model is executed using the text recognition result during the above preset initial interval, and If the above quality level is classified as the above second quality level, it further causes the neural denoising model to be executed without the text recognition result during the above preset initial interval, Electronic device.
13. In processor-implemented methods, Operation (111) of generating guide features by running a neural control model based on an input image; An operation (1120) to generate a current denoised image by executing a neural denoising model based on a previous denoised image, a previous text recognition result, and the guide features; and The operation (1130) of generating a current text recognition result by executing a neural text recognition model based on current denoising features generated by one or more layers of the neural denoising model to generate the above current denoised image. A method including 14. In Paragraph 13, The previously denoised image above The above-mentioned currently denoised image is generated using the above-mentioned neural denoising model at a time step prior to the generation of the above-mentioned currently denoised image, and The above previous text recognition result is A neural text recognition model generated using the neural text recognition model at a time step prior to the generation of the current text recognition result. method.
15. In Paragraph 13 or 14, The one or more layers of the above neural denoising model are Generate the current denoising feature based on the above-mentioned previously denoised image, the above-mentioned previously text recognition result, and the above-mentioned guide feature, and The above neural text recognition model Generating the current text recognition result based on the above current denoising feature, method.
