Electronic device including camera, operation method thereof, and recording medium

Abstract

An operation method of an electronic device according to one embodiment disclosed herein may comprise an operation of acquiring a first image by using a camera of the electronic device. The operation method of the electronic device may comprise an operation of performing a first artificial intelligence operation on the first image by using an NPU of the electronic device. The operation method of the electronic device may comprise an operation of performing a second artificial intelligence operation by using the NPU of the electronic device, on the basis of a result of performing the first artificial intelligence operation. The operation method of the electronic device may comprise an operation of acquiring a composite image for a first image by using the NPU, on the basis of a result of performing the second artificial intelligence operation. The operation of performing the first artificial intelligence operation may comprise an operation of identifying a region of interest from the first image. The operation of performing the first artificial intelligence operation may comprise an operation of identifying an object of interest from the region of interest and acquiring a first partial image corresponding to the object of interest. The operation of performing the second artificial intelligence operation may comprise an operation of applying the first image to a first artificial intelligence neural network model using the NPU of the electronic device and applying the first partial image to a second artificial intelligence neural network model using the NPU. The operation of performing the second artificial intelligence operation may comprise acquiring a second image, output from the first artificial intelligence neural network model by performing image processing on the first image by using the NPU, and a second partial image, output from the second artificial intelligence neural network model by performing image processing on the first partial image, on the basis of a task order input to the NPU from the first artificial intelligence neural network model and the second artificial intelligence neural network model. The operation of acquiring the composite image may comprise an operation of acquiring a third image by compositing the second partial image with the second image.

Description

Electronic device including a camera, method of operation thereof, and recording medium

[0001] The present disclosure relates to an electronic device including a camera, a method of operating the same, and a recording medium.

[0002] With the development of digital technology, various types of electronic devices such as mobile communication terminals, PDAs (personal digital assistants), electronic notebooks, smartphones, tablet PCs (personal computers), or wearable devices are widely used. Electronic devices can provide various functions. For example, electronic devices can provide at least one function by running at least one application in the foreground and / or background.

[0003] The electronic device has a specified operating system (e.g., the Android operating system (Android) TM Various functions can be provided using the operating system. For example, an electronic device can support multiple functions provided using a camera.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art in relation to the present disclosure.

[0005] According to a disclosed embodiment, a method of operation of an electronic device may include an operation of acquiring a first image using a camera of the electronic device. A method of operation of the electronic device may include an operation of performing a first artificial intelligence operation on the first image using an NPU of the electronic device. A method of operation of the electronic device may include an operation of performing a second artificial intelligence operation using an NPU of the electronic device based on the result of performing the first artificial intelligence operation. A method of operation of the electronic device may include an operation of acquiring a synthetic image for the first image using an NPU based on the result of performing the second artificial intelligence operation. An operation of performing the first artificial intelligence operation may include an operation of identifying a region of interest from the first image. An operation of performing the first artificial intelligence operation may include an operation of identifying an object of interest from the region of interest and acquiring a first partial image corresponding to the object of interest. An operation of performing the second artificial intelligence operation may include an operation of applying the first image to a first artificial intelligence neural network model using an NPU of the electronic device and applying the first partial image to a second artificial intelligence neural network model using an NPU. The operation of performing a second artificial intelligence operation may include an operation of obtaining a second partial image output from a second artificial intelligence neural network model by performing image processing on a first image using an NPU based on the task sequence input to the NPU from the first artificial intelligence neural network model and the second artificial intelligence neural network model, and by performing image processing on a first partial image output from the first artificial intelligence neural network model. The operation of obtaining a composite image may include an operation of obtaining a third image by compositing the second partial image with the second image.

[0006] A computer-readable, non-transient recording medium having recorded instructions for controlling an electronic device according to a disclosed embodiment may include a instruction for acquiring a first image using a camera of the electronic device. The recording medium may include a instruction for performing a first artificial intelligence operation on the first image using an NPU of the electronic device. Based on the result of performing the first artificial intelligence operation, the recording medium may include a instruction for performing a second artificial intelligence operation using an NPU of the electronic device. Based on the result of performing the second artificial intelligence operation, the recording medium may include a instruction for acquiring a synthetic image of the first image using an NPU. The instruction for performing the first artificial intelligence operation may include a instruction for identifying a region of interest from the first image. The instruction for performing the first artificial intelligence operation may include a instruction for identifying an object of interest from the region of interest and acquiring a first partial image corresponding to the object of interest. The instruction for performing the second artificial intelligence operation may include a instruction for applying the first image to a first artificial intelligence neural network model using an NPU of the electronic device and applying the first partial image to a second artificial intelligence neural network model using an NPU. The instruction for performing the second artificial intelligence operation may include an instruction for obtaining a second partial image output from the second artificial intelligence neural network model by performing image processing on the first image using the NPU based on the task sequence input to the NPU from the first artificial intelligence neural network model and the second artificial intelligence neural network model, and by performing image processing on the first partial image output from the first artificial intelligence neural network model. The instruction for obtaining the composite image may include an instruction for obtaining a third image by synthesizing the second partial image with the second image.

[0007] An electronic device according to a disclosed embodiment may include a camera, at least one processor including an NPU, and a memory for storing instructions. The electronic device may acquire a first image using the camera. The electronic device may perform a first artificial intelligence operation on the first image using the NPU. Based on the result of performing the first artificial intelligence operation, the electronic device may perform a second artificial intelligence operation using the NPU of the electronic device. Based on the result of performing the second artificial intelligence operation, the electronic device may acquire a synthetic image of the first image using the NPU. As a first artificial intelligence operation, the electronic device may identify a region of interest from the first image. As a first artificial intelligence operation, the electronic device may identify an object of interest from the region of interest. As a first artificial intelligence operation, the electronic device may acquire a first partial image corresponding to the object of interest. As a second artificial intelligence operation, the electronic device may apply the first image to a first artificial intelligence neural network model using the NPU. As a second artificial intelligence operation, the electronic device may apply the first partial image to a second artificial intelligence neural network model using the NPU. The electronic device can obtain a second partial image output from a second artificial intelligence neural network model by performing image processing on a first image using the NPU based on the task sequence input to the NPU from the first artificial intelligence neural network model and the second artificial intelligence neural network model as a second artificial intelligence operation, and by performing image processing on a second image output from the first artificial intelligence neural network model and a first partial image. The electronic device can obtain a third image by synthesizing the second partial image with the second image.

[0008] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0009] FIG. 1 is a block diagram illustrating an electronic device in a network environment according to various embodiments.

[0010] FIG. 2 is a block diagram illustrating a camera module according to various embodiments.

[0011] FIG. 3 is a block diagram illustrating the configuration of an electronic device according to one embodiment.

[0012] FIG. 4 is a block diagram illustrating the configuration of an electronic device according to one embodiment.

[0013] FIG. 5 is a diagram illustrating an embodiment in which an electronic device according to one embodiment detects a region of interest using an artificial intelligence neural network model.

[0014] FIG. 6 is a diagram illustrating an embodiment in which an electronic device according to one embodiment analyzes a region of interest using an artificial intelligence neural network model.

[0015] FIG. 7 is a diagram illustrating an embodiment in which an electronic device according to one embodiment improves an image using an artificial intelligence neural network model.

[0016] FIG. 8 is a diagram illustrating an embodiment in which an electronic device according to one embodiment improves a region of interest using an artificial intelligence neural network model.

[0017] FIG. 9 is a diagram illustrating an embodiment in which an electronic device according to one embodiment synthesizes an improved region of interest into an image.

[0018] FIG. 10 is a flowchart relating to a method of operation of an electronic device according to one embodiment.

[0019] FIG. 11 is a diagram illustrating the flow of operation of an electronic device according to one embodiment.

[0020] FIG. 12 is a drawing showing a system including a generative artificial intelligence model according to one embodiment.

[0021] The electronic device can acquire images through a camera module. The electronic device can perform post-processing on the acquired images. For example, the electronic device can perform at least one processing such as white balance adjustment, contrast adjustment, saturation adjustment, color correction, sharpening, noise removal, tone mapping, and edge enhancement on the acquired images using an artificial intelligence neural network model.

[0022] According to one embodiment, an electronic device can improve an image using an artificial intelligence neural network model. For example, the electronic device can improve the entire image using an artificial intelligence neural network model. For example, the electronic device can improve a region of interest within the image using an artificial intelligence neural network model. For example, the electronic device can improve the region of interest according to a set value. For example, the electronic device can obtain a final image by synthesizing the improved image and the region of interest.

[0023] The technical problems to be solved in this document are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which the present invention belongs from the description in this disclosure.

[0024] Hereinafter, embodiments are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present invention. However, the disclosed embodiments may be implemented in various different forms and are not limited to the embodiments described herein.

[0025] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (60), sound output module (65), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0026] The processor (120) can control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., a program (140)) and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0027] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0028] The number of processors (120) may be one or more. For example, the processor (120) may have the structure of a multi-core processor such as a dual core, a quad core, or a hexa core.

[0029] The processor (120) can control the operations of the electronic device (101) by executing instructions stored in memory (130). For example, the processor (120) may correspond to a plurality of processors that divide and collectively perform a plurality of operations among the processors.

[0030] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., a processor (120) or a sensor module (176)). The data may include, for example, software (e.g., a program (140)) and input data or output data for related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0031] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0032] The input module (60) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (60) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0033] The sound output module (65) can output a sound signal to the outside of the electronic device (101). The sound output module (65) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0034] The display module (160) can visually provide information to the outside of the electronic device (101) (e.g., a user). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0035] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (60) or output sound through the sound output module (65) or an external electronic device (e.g., electronic device (102)) (e.g., a speaker or headphones) that is directly or wirelessly connected to the electronic device (101).

[0036] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0037] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0038] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0039] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0040] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0041] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0042] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0043] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0044] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. The NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate for eMBB realization (e.g., 20 Gbps or more), loss coverage for mMTC realization (e.g., 164 dB or less), or U-plane latency for URLLC realization (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less).

[0045] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., array antennas). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0046] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0047] At least some of the above components may be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and may exchange signals (e.g., commands or data) with each other.

[0048] According to one embodiment, commands or data may be transmitted or received between an electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0049] FIG. 2 is a block diagram relating to an electronic device (101) including a camera module (180) according to various embodiments. The electronic device (101) of FIG. 2 may correspond to the electronic device (101) described with reference to FIG. 1. Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260).

[0050] The lens assembly (210) can collect light emitted from a subject that is the subject of image capture. The lens assembly (210) may include one or more lenses.

[0051] According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties different from the lens properties of other lens assemblies. The lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

[0052] The flash (220) may emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) may include one or more light-emitting diodes (e.g., RGB (red-green-blue) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp.

[0053] The image sensor (230) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) may include, for example, one image sensor selected from image sensors with different attributes such as an RGB sensor, a BW (black and white) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same attribute, or a plurality of image sensors having different attributes. Each image sensor included in the image sensor (230) may be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0054] The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction in response to the movement of the camera module (180) or the electronic device (101) including it, or control the operational characteristics of the image sensor (230) (e.g., adjust read-out timing). This allows for compensating for at least some of the negative effects caused by the movement on the image being captured.

[0055] According to one embodiment, the image stabilizer (240) can detect the movement of the camera module (180) or the electronic device (101) using a gyroscope sensor (not shown) or an accelerometer sensor (not shown) placed inside or outside the camera module (180). For example, the image stabilizer (240) may be implemented as an optical image stabilizer.

[0056] The memory (250) may temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing operation. For example, if image acquisition by the shutter is delayed or multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and the corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Subsequently, when a specified condition is satisfied (e.g., user input or system command), at least a portion of the original image stored in the memory (250) may be acquired and processed by, for example, an image signal processor (260). According to one embodiment, the memory (250) may be configured as at least a portion of the memory (130) or as a separate memory that operates independently thereof.

[0057] According to one embodiment, the memory (250) can store images acquired and output as preview images at least temporarily. The preview image may include an image provided by an electronic device (101) that allows the user to check the location, lighting, or composition of the object to be photographed so that the user can acquire the desired image.

[0058] The image signal processor (260) can perform one or more image processing operations on an image obtained through the image sensor (230) or an image stored in memory (250). The above one or more image processing methods may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softing). Additionally or generally, the image signal processor (260) may perform control (e.g., exposure time control, or readout timing control) over at least one of the components included in the camera module (180) (e.g., image sensor (230)). The image processed by the image signal processor (260) may be stored back in memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) is at least part of the processor (120). It may be configured as a separate processor that operates independently of the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) as is or after additional image processing by the processor (120).

[0059] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180) each having different attributes or functions. For example, among the plurality of camera modules (180), at least one may be a wide-angle camera and at least another may be a telephoto camera. Similarly, among the plurality of camera modules (180), at least one may be a front camera and at least another may be a rear camera.

[0060] FIG. 3 is a block diagram illustrating the configuration of an electronic device (101) according to one embodiment.

[0061] Referring to FIG. 3, an electronic device (101) according to one embodiment may include a camera module (380), a memory (330), and a processor (320). An electronic device (101) according to one embodiment may further include a display (360). The display (360) may be replaced by an external display connected to the electronic device (101). The electronic device (101), processor (320), memory (330), camera module (380), and display (360) may correspond to the electronic device (101), processor (120), memory (130), camera module (180), and display module (160) described above with reference to FIG. 1 and FIG. 2, respectively. However, the components of the electronic device (101) shown in FIG. 3 are for the purpose of explaining one embodiment, and the electronic device (101) may include more components than those shown in FIG. 3, or may include other components that can replace at least some of the components. For example, the memory (330) is not limited to a storage medium included in the electronic device (101) and may include a cloud storage outside the electronic device (101).

[0062] According to one embodiment, the camera module (380) may include a lens unit (381) (e.g., lens assembly (210) of FIG. 2) comprising at least one lens for collecting light, and an image sensor (383) (e.g., image sensor (230) of FIG. 2) for converting a light signal passing through the lens unit (381) into a digital signal.

[0063] The image sensor (383) may include a color filter comprising a plurality of light-receiving elements, a plurality of micro-lenses, and a plurality of color channels. The plurality of light-receiving elements may include photodiodes arranged in an array having M rows and N columns to correspond to one micro-lens. Here, M and N may each be natural numbers greater than or equal to 1.

[0064] The color filter included in the image sensor (383) may be composed of a non-Bayer pattern (e.g., a tetra pattern, a nona pattern, a hexadeca pattern). The colors of the element groups of the color filter composed of a non-Bayer pattern may be matched to correspond to the Bayer pattern.

[0065] According to one embodiment, the image sensor (383) can output an image signal (e.g., raw image data) composed of a non-Bayer pattern. For example, the image sensor (383), which includes a color filter composed of a non-Bayer pattern, can output an image signal composed of a non-Bayer pattern by determining a pixel value using the output value of a light-receiving element corresponding to an element. The image signal output from the image sensor (383) may be data in which the color pattern is maintained, as the color order of the color pattern of the image sensor (383) is not changed.

[0066] According to one embodiment, the image sensor (383) can output an image signal (e.g., raw image data) composed of a Bayer pattern. For example, the image sensor (383) including a color filter composed of a non-Bayer pattern can output an image signal composed of a Bayer pattern by binning the output values ​​of light-receiving elements corresponding to the element groups constituting the color filter.

[0067] According to one embodiment, the image sensor (383) may operate in a high-resolution mode or a crop mode. The high-resolution mode may include a mode in which each of the output values ​​of the light-receiving elements included in the image sensor (383) is used as the pixel value of a pixel corresponding to each of the light-receiving elements. In the present disclosure, the term "pixel" may mean the smallest unit constituting a digital image. The resolution of an image may be expressed as the number of pixels included in the image. For example, if an image consists of axb pixels arranged in a row and b columns, the resolution of the image may be indicated as axb. For example, an electronic device (101) may acquire high-resolution image data by using an image sensor (383) composed of 50 megapixels (Mp) of light-receiving elements and using each of the output values ​​of the 50 megapixels of light-receiving elements as the pixel value of each of the pixels corresponding to the light-receiving elements. The high-resolution mode may be understood as a full pixel mode.

[0068] The crop mode may include a mode in which the output values ​​of a predetermined number of light-receiving elements (for example, a predetermined number of light-receiving elements located in the center of the image sensor) among the light-receiving elements are used as pixel values. The electronic device (101) can acquire an image with a narrowed field of view (FOV) through the crop mode and can provide the user with an experience similar to that of a zoom-in function. For example, when the electronic device (101) operates in a crop mode in which the output values ​​of 12.5 Mp light-receiving elements located in the center of an image sensor (383) composed of 50 Mp light-receiving elements are used as pixel values, the user can be provided with an experience similar to that of a 2x zoom-in function.

[0069] According to one embodiment, the image sensor (383) may operate in a low-light mode or a multi-frame composite mode. The low-light mode may include a mode in which the pixel values ​​of a first pixel corresponding to the first light-receiving elements are used based on values ​​output from first light-receiving elements corresponding to elements of a color filter matched to the same color of the light-receiving elements included in the image sensor (383). The multi-frame composite mode may include a mode in which the image sensor (383) acquires a plurality of frames in which there is a difference in exposure values.

[0070] According to one embodiment, the memory (330) can store instructions that can be executed by the processor (320). By executing the instructions stored in the memory (330), the processor (320) can perform operations or control components of the electronic device (101).

[0071] In the present disclosure, the operation of the electronic device (101) may be understood as being performed by at least one processor (320) executing instructions. According to one embodiment, the processor (320) may include at least one of an application processor (AP), a central processing unit (CPU), an image signal processor (ISP) (e.g., the image signal processor (260) of FIG. 2), a graphical processing unit (GPU), or a neural processing unit (NPU). For example, at least one processor (320) may include an application processor. At least one processor (320) may acquire image data based on an image frame containing information read out from an image sensor (383).

[0072] According to one embodiment, at least one processor (320) can control a camera module (380) by executing a camera application. For example, the processor (320) can initiate the operation of the camera module (380) through the camera application. For example, the processor (320) can provide a request to the camera module (380) through the camera application for the acquisition of at least one frame (e.g., a frame for preview, a frame for capture). For example, the processor (320) can acquire one or more frames through the camera application. For example, the processor (320) can identify user input received through the camera application (e.g., user input regarding the acquisition of a capture image). For example, the capture image may include an image in which one or more preset operations are applied to an image signal acquired by the camera module (380). For example, the captured image may include an image to which at least one operation has been applied, such as demosaicing, white balance, contrast, saturation value adjustment, gamma correction, color correction, sharpening, noise removal, tone mapping, and edge enhancement. For example, the processor (320) may acquire image data (e.g., preview image data, draft image data, captured image data, video data) through a camera application. For example, the processor (320) may display the acquired image data using a display (360).

[0073] According to one embodiment, at least one processor (320) can transmit a control signal to an image sensor (383) to perform a read operation. For example, at least one processor (320) can transmit the control signal based on a specified communication method (e.g., I2C (inter-integrated circuit), I3C (improved inter-integrated circuit)). At least one processor (320) can acquire image data output from the image sensor (383). The image data may include image frames. For example, at least one processor (320) can receive image data through an interface connected to the image sensor (383) (e.g., MIPI (mobile industry processor interface)).

[0074] According to one embodiment, at least one processor (320) can control an image sensor (383) to acquire an image corresponding to a user input. The processor (320) can control the image sensor (383) to operate based on a selected shooting mode. For example, the image sensor (383) can output an image signal (e.g., raw image data) by reading out each output of a light receiving element constituting the image sensor (383) based on a first shooting mode (e.g., high-resolution shooting mode) so that each output corresponds to a pixel corresponding to each of the light receiving elements. For example, the image sensor (383) can output an image signal (e.g., raw image data) by reading out each output of a predetermined number of first light receiving elements among a plurality of light receiving elements based on a second shooting mode (e.g., crop shooting mode) so that each output corresponds to a pixel corresponding to each of the first light receiving elements. For example, the image sensor (383) can output an image signal (e.g., raw image data) by reading out a first group of light-receiving elements based on a third shooting mode (e.g., low-light shooting mode) such that the output of the first group corresponds to a first pixel corresponding to the first group. For example, the image sensor (383) can output a short exposure image signal and a long exposure image signal by controlling the exposure time of the image sensor (383) based on a fourth shooting mode (e.g., HDR shooting mode) so that there is a preset difference in exposure.

[0075] According to one embodiment, at least one processor (320) can acquire image data by performing operations on an image signal (e.g., raw image data) output from an image sensor (383). For example, the processor (320) can perform at least one operation on the image signal (e.g., demosaicing (or Bayer interpolation), an operation to adjust white balance, contrast, and saturation values, gamma correction, brightness correction, color correction, sharpening, noise removal, tone mapping, edge enhancement).

[0076] For example, the processor (320) can generate a second image signal (for example, raw image data composed of a Bayer pattern) by performing remosaicing on a first image signal output from an image sensor (383) (for example, raw image data composed of a non-Bayer pattern). For example, the processor (320) can perform at least one operation (for example, demosaicing, white balance, contrast, saturation value adjustment operation, gamma correction, brightness correction, color correction, sharpening, noise removal, tone mapping, edge enhancement) by inputting the image signal output from the image sensor (383) to an image signal processor (for example, the image signal processor (260) of FIG. 2). The processor (320) can store the generated image data in memory (330).

[0077] For example, the processor (320) can identify objects included in an image using an image signal. For example, the processor (320) can identify objects included in an image by performing an object recognition operation on the image signal. For example, the processor (320) can identify a main object and a sub-object among the objects included in the image. For example, the processor (320) can identify a face included in an image by performing face recognition.

[0078] According to one embodiment, at least one processor (320) can initiate the operation of a camera module (380). For example, the processor (320) can initiate the operation of the camera module (380) by providing a control signal to a hardware abstraction layer (HAL) that includes the identification of the camera and a command to initiate the operation of the camera.

[0079] According to one embodiment, at least one processor (320) can acquire one or more frames. For example, the processor (320) can acquire one or more frames by controlling the camera module (380) through a request to acquire frames. For example, the processor (320) can acquire a preview frame composed of a Bayer pattern. For example, the processor (320) can acquire a capture frame composed of a non-Bayer pattern.

[0080] According to one embodiment, at least one processor (320) can convert a first image signal composed of a first color pattern (e.g., a non-Bayer pattern) into a second image signal composed of a second color pattern (e.g., a Bayer pattern). For example, the processor (320) can convert the first image signal into a second image signal by remosaicizing it. For example, the processor (320) can convert the first image signal into a second image signal by binning it. For example, the processor (320) can control the processing unit of the image sensor (383) to convert the first image signal into a second image signal.

[0081] According to one embodiment, at least one processor (320) can identify received user input. For example, the processor (320) can identify user input changing a shooting mode (e.g., high resolution mode, crop mode, low light mode, multi-frame composite mode, still image shooting mode, video shooting mode). For example, the processor (320) can identify user input regarding the acquisition of still images and / or videos. For example, the processor (320) can identify user input requesting the acquisition of a multi-frame composite image.

[0082] According to one embodiment, at least one processor (320) can acquire preview image data. For example, the processor (320) can generate preview image data by applying an image signal composed of a Bayer pattern output from an image sensor (383) including a Bayer pattern color filter to an image signal processor (260). For example, the processor (320) can control the processing unit of the image sensor (383) so that an image signal composed of a Bayer pattern is output by binning the output values ​​of the light receiving elements of the image sensor (383) including a color filter of a non-Bayer pattern (e.g., tetra, nonar, hexadeca). For example, the tetra pattern may be a non-Bayer pattern in which a 2x2 array of light receiving elements corresponds to a color filter of the same color. For example, the nonar pattern may be a non-Bayer pattern in which a 3x3 array of light receiving elements corresponds to a color filter of the same color. For example, a hexadecca pattern may be a non-Bayer pattern in which a 4x4 array of light-receiving elements corresponds to a color filter of the same color.

[0083] According to one embodiment, at least one processor (320) can acquire a capture image. For example, the processor (320) can generate a capture image by performing at least one operation (e.g., demosaicing, white balance, contrast, saturation value adjustment operation, gamma correction, color correction, sharpening, noise removal, tone mapping, edge enhancement) on a first image signal composed of a non-Bayer pattern output from an image sensor (383). For example, the processor (320) can generate a capture image by performing an operation on an image signal of a frame composed of a non-Bayer pattern based on a high-resolution mode. For example, the processor (320) can generate a capture image by performing an operation on an image signal of a frame composed of a non-Bayer pattern based on a crop mode.

[0084] According to one embodiment, at least one processor (320) can display acquired image data using a display (360). For example, the processor (320) can display at least one of preview image data and captured image data in at least a part area of ​​a camera application.

[0085] According to one embodiment, the display (360) can display one or more pieces of information through the control of at least one processor (320). For example, the display (360) can display a user interface (UI) of an electronic device. For example, the display (360) can display an execution screen of an application running on an electronic device. For example, the display (360) can display preview image data and captured image data.

[0086] FIG. 4 is a block diagram illustrating the configuration of an electronic device (101) according to one embodiment. The electronic device (101) of FIG. 4 may correspond to the electronic device described with reference to FIG. 1 to FIG. 3 (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 2, and the electronic device (101) of FIG. 3).

[0087] According to one embodiment, the electronic device (101) may include a memory (430) (e.g., memory (130) of FIG. 1) and an application processor (400) (e.g., processor (120) of FIG. 1). The application processor (400) may include a central processing unit (CPU) (420) and a neural network processing unit (NPU) (410). The central processing unit (420) may be configured to perform complex operations serially more than the neural network processing unit (410). The neural network processing unit (410) may include a device configured to efficiently process parallel operations to process a neural network model. The neural network processing unit (410) may be configured to perform a plurality of simple operations in parallel more than the central processing unit (420).

[0088] According to one embodiment, the neural network processing unit (410) and the central processing unit (420) may be integrated into the application processor (400). The application processor (400) may be configured so that the neural network processing unit (410) and the central processing unit (420) can communicate directly.

[0089] According to one embodiment, a neural network processing unit (410) may be connected to an interconnect (430). The interconnect (430) may be configured to directly connect the neural network processing unit (410) to an arbitrator (421) of a central processing unit (420). The central processing unit (420) may include an arbitrator (421) and a cache memory (423). The arbitrator (421) may be configured to selectively connect the cache memory (423) to either a processing circuit or the interconnect (430) for the central processing unit (420) to process data.

[0090] According to one embodiment, the neural network processing unit (410) and the central processing unit (420) can directly store data in the cache memory (423) or retrieve data stored in the cache memory (423) through the arbitrator (421). Since the neural network processing unit (410) can directly access the cache memory (423) of the central processing unit (420) through the arbitrator (421) and the interconnector (430), the delay caused by DRAM connection can be reduced. The neural network processing unit (410) or the central processing unit (420) can exchange data by storing data in the cache memory (423) or retrieving data from the cache memory (423).

[0091] According to one embodiment, when a plurality of tasks are input, the neural network processing device (410) can process the neural network model in the order of the input tasks. The neural network processing device (410) can process the artificial intelligence neural network model corresponding to the input tasks.

[0092] According to one embodiment, the neural network processing device (410) may request a task for a second inference operation to improve the quality of a first portion of an image while a task for a first inference operation to improve the quality of a first portion of an image is in progress. For example, the neural network processing device (410) may perform the second inference operation after the first inference operation is completed. For example, the second inference operation may be queued while the first inference operation is in progress. For example, the task may include a pipelined process as an execution unit of a resource with a distinct role within the neural network processing device (410).

[0093] According to one embodiment, the neural network processing unit (410) may execute an artificial intelligence neural network model corresponding to the task in order to process the input task. The neural network processing unit (410) may frequently change the artificial intelligence neural network model being executed, which may result in significant time consumption due to overhead. Depending on the process scheduling, the neural network processing unit (410) may experience unpredictable performance degradation due to overhead.

[0094] According to one embodiment, the neural network processing unit (410) can prioritize processing tasks with low computational load (SJF, shortest job first). For example, the neural network processing unit (410) can prioritize processing tasks with low computational load, such as a task to identify a region of interest from an image, a task to determine whether to perform image processing on a region of interest, or a task to parse an object of interest from a region of interest. For example, the neural network processing unit (410) can prioritize processing tasks with low computational load over tasks with high computational load, such as an inference computation task for image processing of a region of interest and an inference computation task for image processing of an entire image.

[0095] According to one embodiment, an application processor (400) can process tasks using multi-threading and pipelining. For example, the application processor (400) can process tasks by configuring a pipeline for each execution code when the task includes execution code that can be processed by a central processing unit (420) and execution code that can be processed by a neural network processing unit (410).

[0096] FIG. 5 is a diagram illustrating an embodiment in which an electronic device according to one embodiment detects a region of interest using an artificial intelligence neural network model. The electronic device (101) of FIG. 5 may correspond to the electronic device (101) described with reference to FIG. 1 to FIG. 4. The neural network processing unit (NPU) (500) of FIG. 5 may correspond to the neural network processing unit (NPU) (410) described with reference to FIG. 4.

[0097] According to one embodiment, an electronic device (101) can acquire a first image (510). For example, the electronic device (101) can acquire the first image (510) using a camera module (e.g., the camera module (180) of FIG. 1). For example, the electronic device (101) can acquire the first image (510) by controlling the camera module (e.g., the camera module (180) of FIG. 1) to take a picture by running a camera application. For example, the electronic device (101) can acquire the first image (510) stored in a memory (e.g., the memory (130) of FIG. 1). For example, the electronic device (101) can download the first image (510).

[0098] According to one embodiment, the electronic device (101) can perform an operation to identify a region of interest (520) from a first image (510) using a neural network processing device (500). For example, the neural network processing device (500) can identify the region of interest (520) using a first artificial intelligence neural network model (501). For example, the first artificial intelligence neural network model (501) may be trained to identify a region of interest (520) from an input first image (510). For example, the first artificial intelligence neural network model (501) may be trained to output the coordinates of a region of interest (521, 522, 523, 524) from the first image (510). For example, the first artificial intelligence neural network model (501) may be trained to output the coordinates of a region of interest (521, 522, 523, 524) using a percentage range of the first image (510). For example, the first artificial intelligence neural network model (501) may be trained to output the coordinates of the region of interest (521, 522, 523, 524) using absolute coordinates.

[0099] FIG. 6 is a diagram illustrating an embodiment in which an electronic device (e.g., the electronic device (101) of FIG. 1) according to one embodiment analyzes a region of interest (520) using an artificial intelligence neural network model. FIG. 6 is a diagram illustrating an embodiment in which the electronic device infers about the region of interest (520) described with reference to FIG. 5 using a neural network processing unit (NPU) (600). The neural network processing unit (NPU) (600) of FIG. 6 may correspond to the neural network processing unit (NPU) (410) described with reference to FIG. 4.

[0100] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 5) can perform inference operations on a region of interest (520) using a neural network processing device (600). For example, the electronic device can analyze regions of interest (521, 522, 523, 524) and perform inference operations to identify whether image processing is required for each of the regions of interest (521, 522, 523, 524). For example, the electronic device can perform inference operations to parse an object of interest (602) from regions of interest (521, 522) where image processing is required.

[0101] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 5) can enable a neural network processing device (600) to perform inference operations on regions of interest (521, 522, 523, 524) through a region of interest analysis module (610). For example, the electronic device can input a region of interest (601) that has been resized to fit the kernel size of a second artificial intelligence neural network model (611) to the second artificial intelligence neural network model (611) through the region of interest analysis module (610).

[0102] For example, the neural network processing unit (600) can identify regions of interest (523, 524) that do not require image processing by analyzing regions of interest (521, 522, 523, 524) using a second artificial intelligence neural network model (611). For example, the neural network processing unit (600) can identify regions of interest (523, 524) that are smaller than a predetermined size as regions of interest that do not require image processing by analyzing the size of regions of interest (521, 522, 523, 524). For example, the neural network processing unit (600) can identify regions of interest (523, 524) that are of low interest as regions of interest that do not require image processing by analyzing the location of regions of interest (521, 522, 523, 524).

[0103] For example, the neural network processing unit (600) can perform an inference operation to obtain analysis data of the region of interest (601) by analyzing the region of interest (601) using a second artificial intelligence neural network model (611). For example, the neural network processing unit (600) can perform an inference operation to obtain analysis data by analyzing the size, clarity, brightness, noise, and defined aesthetic elements of the region of interest (601) according to defined criteria. For example, the neural network processing unit (600) can perform an inference operation to obtain analysis data by analyzing the degree of similarity between a first feature (e.g., wrinkles of a person) identified from an object of interest (602) included in the region of interest (601) and a set second feature (e.g., depth, size, and location of wrinkles). For example, the second artificial intelligence neural network model (611) may be trained to output analysis data containing higher values ​​as the result of analyzing the region of interest (601) conforms to the defined criteria. For example, the second artificial intelligence neural network model (611) may be trained to output analysis data containing lower values ​​as the result of analyzing the region of interest (601) differs from a defined standard. For example, the analysis data may contain analysis values ​​greater than or equal to 0.0.

[0104] According to one embodiment, the electronic device can identify whether the region of interest (601) requires image processing based on analysis data of the region of interest (601). For example, the electronic device can identify whether the region of interest (601) requires image processing by using a region of interest analysis module (610). For example, the electronic device can identify that the region of interest (601) requires image processing based on identifying that the analysis value included in the analysis data of the region of interest (601) is less than a defined threshold value. For example, the electronic device can determine whether to perform image processing on the region of interest (601) containing the object of interest (602) based on at least one of the brightness of the object of interest (602), noise included in the region corresponding to the object of interest (602), or the similarity between a first feature identified from the object of interest (602) (e.g., wrinkles of a person) and a set second feature (e.g., depth, size, location of wrinkles). For example, the electronic device may identify that, for the region of interest (601), at least one of image processing to improve clarity, image processing to adjust brightness, image processing to remove noise, and image processing to improve aesthetic elements according to defined settings is required.

[0105] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 5) can enable a neural network processing device (600) to perform inference operations on a region of interest (601) through an object of interest parsing module (620). For example, in response to identification that image processing is required for a region of interest (601), the neural network processing device (600) can perform inference operations to parse an object of interest (602) included in the region of interest (601) using a third artificial intelligence neural network model (621) and the object of interest parsing module (620). For example, the third artificial intelligence neural network model (621) may be trained to parse a region corresponding to an object of interest (602) included in the region of interest (601) in response to inputting a region of interest (601). For example, the third artificial intelligence neural network model (621) may be trained to output a probability that each part of the region of interest (601) of an array having M rows and N columns corresponds to an object of interest (602). For example, the third artificial intelligence neural network model (621) may be trained to output a probability value that is expressed as a floating-point number between 0.0 and 1.0 for each part of the region of interest (601). For example, the third artificial intelligence neural network model (621) may be trained to output a matrix (692) containing the probabilities of the parts of the region of interest (601). For example, the third artificial intelligence neural network model (621) may output the probability that corresponds to an object of interest (602) in a batch size of the third artificial intelligence neural network model (621). For example, the electronic device can use the matrix (692) as a blur array used for alpha blending by converting the output data of the third artificial intelligence neural network model (621) of the batch size of the third artificial intelligence neural network model (621) to correspond to the regions of interest (521, 522).

[0106] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 5) may acquire object-oriented data (690) regarding regions of interest (521, 522). For example, the object-oriented data may be structured data such that detailed information regarding the regions of interest can be processed as a single object. For example, the object-oriented data (690) may include the coordinates of the regions of interest (521, 522) within the first image (510). For example, the object-oriented data (690) may include x-axis coordinate values ​​(e.g., Xstart, Xend) and y-axis coordinate values ​​(e.g., Ystart, Yend) of the regions of interest (521, 522). For example, the object-oriented data (690) may include image data (691) corresponding to the regions of interest (521, 522). For example, the objectification data (690) may include a matrix (692) for regions of interest (521, 522). For example, the objectification data (690) may include shooting information (e.g., aperture value, shutter speed, sensitivity value, color temperature value) of a camera module (e.g., camera module (180) of FIG. 1) when the first image (510) is captured. For example, the objectification data (690) may include information regarding the size of buffer data for image processing on regions of interest (521, 522). For example, the objectification data (690) may be stored in memory (e.g., memory (130) of FIG. 1).

[0107] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 5) can perform inference operations on all regions of interest (521, 522) detected from a first image (510). For example, the electronic device can perform an inference operation to identify whether image processing is required for the first region of interest (521) and an inference operation to parse the object of interest (602). For example, the electronic device can perform an inference operation to identify whether image processing is required for the second region of interest (522) and an inference operation to parse the object of interest (602). For example, the electronic device (e.g., the electronic device (101) of FIG. 5) can perform inference operations sequentially on all regions of interest (521, 522) detected from the first image (510). For example, the electronic device can perform an inference operation for a second region of interest (522) after the inference operation for a first region of interest (521) is completed.

[0108] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 5) may process the inference operations described with reference to FIG. 6 in priority over the inference operations described with reference to FIG. 7 and FIG. 8.

[0109] FIG. 7 is a diagram illustrating an embodiment in which an electronic device (e.g., the electronic device (101) of FIG. 1) according to one embodiment improves an image using an artificial intelligence neural network model. FIG. 7 is a diagram illustrating an embodiment in which the electronic device infers about the first image (510) described with reference to FIG. 5 using a neural network processing unit (NPU) (700). The neural network processing unit (NPU) (700) of FIG. 7 may correspond to the neural network processing unit (NPU) (410) described with reference to FIG. 4.

[0110] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 5) can perform inference operations on a first image (510) using a neural network processing device (700). For example, the neural network processing device (700) can perform inference operations on the first image (510) using a fourth artificial intelligence neural network model (701). For example, the electronic device can divide the first image (510) into a plurality of parts (710) corresponding to the size of the batch of the fourth artificial intelligence neural network model (701). For example, the electronic device can sequentially input the plurality of parts (710) to the fourth artificial intelligence neural network model (701).

[0111] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 5) may allocate memory corresponding to the first image (510) to an inference operation. For example, the electronic device (e.g., the electronic device (101) of FIG. 5) may allocate buffer memory required for image processing of the first image (510) to an inference operation for the first image (510).

[0112] According to one embodiment, the neural network processing device (700) can perform inference operations to improve the first image (510) using the fourth artificial intelligence neural network model (701). For example, the fourth artificial intelligence neural network model (701) may be trained to analyze the first image (510) and apply at least one image processing. For example, the fourth artificial intelligence neural network model (701) may be trained to remove noise included in the image data. For example, the fourth artificial intelligence neural network model (701) may be trained to improve the clarity of the image data. For example, the fourth artificial intelligence neural network model (701) may be trained to adjust the color of the image data. For example, the fourth artificial intelligence neural network model (701) may be trained to adjust the brightness of the image data. For example, the fourth artificial intelligence neural network model (701) may be trained to adjust the contrast of the image data. For example, the fourth artificial intelligence neural network model (701) may be trained to improve the detail of the image data. For example, the fourth artificial intelligence neural network model (701) may be trained to identify and remove unnecessary objects from image data.

[0113] According to one embodiment, the neural network processing device (700) can perform at least one inference operation for image processing on a first part (711) input to the fourth artificial intelligence neural network model (701). For example, the neural network processing device (700) can perform an inference operation for at least one of image processing to remove noise included in the first part (711), image processing to improve the clarity of the first part (711), image processing to adjust the color of the first part (711), image processing to adjust the brightness of the first part (711), image processing to adjust the contrast of the first part (711), image processing to improve the detail of the first part (711), or image processing to identify and remove unnecessary objects from the first part (711). For example, the fourth artificial intelligence neural network model (701) can output a first' part (720) to which image processing has been applied to the first part (711) in response to the input of the first part (711). For example, the electronic device can store the first' portion (720) output from the fourth artificial intelligence neural network model (701). For example, the electronic device can temporarily store the first' portion (720) in an allocated buffer memory.

[0114] For example, the neural network processing unit (700) can sequentially perform inference operations on a plurality of parts (710) using the fourth artificial intelligence neural network model (701). For example, the electronic device can obtain a second image (730) from image data output from the fourth artificial intelligence neural network model (701). For example, the electronic device can obtain the second image (730) by combining data to which image processing has been applied to a plurality of parts (710) that are sequentially output from the fourth artificial intelligence neural network model (701). For example, the electronic device can obtain the second image (730) by sequentially storing image data output sequentially from the fourth artificial intelligence neural network model (701) in a buffer memory.

[0115] FIG. 8 is a diagram illustrating an embodiment in which an electronic device according to one embodiment improves a region of interest using an artificial intelligence neural network model. FIG. 8 is a diagram illustrating an embodiment in which an electronic device infers about a region of interest (520) described with reference to FIG. 5 using a neural network processing unit (NPU) (800). The neural network processing unit (NPU) (800) of FIG. 8 may correspond to the neural network processing unit (NPU) (410) described with reference to FIG. 4.

[0116] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 5) can acquire a first partial image (810). For example, the electronic device can acquire a first partial image (810) corresponding to a region of interest (e.g., the region of interest (521) of FIG. 5) for which image processing is determined to be required. For example, the electronic device can acquire the first partial image (810) based on objectification data (e.g., the objectification data (690) of FIG. 6). For example, the electronic device can acquire the first partial image (810) from the first image (510) based on the coordinates of the region of interest (521) included in the objectification data (e.g., the objectification data (690) of FIG. 6). For example, the electronic device can acquire the first partial image (810) from image data (691) included in the objectification data (e.g., the objectification data (690) of FIG. 6).

[0117] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 5) can perform inference operations on a first partial image (810) using a neural network processing device (800). For example, the neural network processing device (800) can perform inference operations on the first partial image (810) using a fifth artificial intelligence neural network model (801). For example, the fifth artificial intelligence neural network model (801) may be trained to remove noise included in the image data. For example, the fifth artificial intelligence neural network model (801) may be trained to improve the clarity of the image data. For example, the fifth artificial intelligence neural network model (801) may be trained to adjust the color of the image data. For example, the fifth artificial intelligence neural network model (801) may be trained to adjust the brightness of the image data. For example, the fifth artificial intelligence neural network model (801) may be trained to adjust the contrast of the image data. For example, the fifth artificial intelligence neural network model (801) may be trained to enhance the detail of the image data. For example, the fifth artificial intelligence neural network model (801) may be trained to identify and remove unnecessary objects from the image data. For example, the fifth artificial intelligence neural network model (801) may be trained to modify the image data so that aesthetic elements included in the image data (e.g., wrinkles on a person, blemishes on the skin, makeup) meet defined criteria. For example, the fifth artificial intelligence neural network model (801) may be trained to adjust the sharpness of high-frequency regions of the image data (e.g., hair).

[0118] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 5) may provide a first partial image (810) to a fifth artificial intelligence neural network model (801). For example, the electronic device may resize the first partial image (810) to correspond to the size of a batch of the fifth artificial intelligence neural network model (801) and input it to the fifth artificial intelligence neural network model (801).

[0119] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 5) may dynamically allocate memory for inference operations. For example, the electronic device may allocate memory for inference operations based on the number of regions of interest (e.g., regions of interest (521, 522) of FIG. 5). For example, the electronic device may dynamically allocate memory based on information regarding the size of buffer data for image processing included in objectification data (e.g., objectification data (690) of FIG. 6) corresponding to each of the regions of interest (e.g., regions of interest (521, 522) of FIG. 5).

[0120] According to one embodiment, the neural network processing device (800) can perform an inference operation to improve the first partial image (810) input to the fifth artificial intelligence neural network model (801). For example, the neural network processing device (800) can perform an inference operation for at least one of image processing to remove noise included in the first partial image (810), image processing to improve the sharpness of the first partial image (810), image processing to adjust the color of the first partial image (810), image processing to adjust the brightness of the first partial image (810), image processing to adjust the contrast of the first partial image (810), image processing to improve the detail of the first partial image (810), image processing to identify and remove unnecessary objects from the first partial image (810), image processing to modify aesthetic elements (e.g., wrinkles of a person, blemishes on the skin, makeup) included in the first partial image (810) to conform to defined standards, or image processing to adjust the sharpness of a high-frequency region (e.g., hair) of the first partial image (810). For example, the fifth artificial intelligence neural network model (801) can output a second partial image (820) to which image processing has been applied to the first partial image (810) in response to the input of the first partial image (810). For example, an electronic device can store the second partial image (820) output from the fifth artificial intelligence neural network model (801). For example, the electronic device can temporarily store the second partial image (820) in an allocated buffer memory.

[0121] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 5) can sequentially perform inference operations on all partial images corresponding to a region of interest (e.g., the region of interest (521) of FIG. 5) for which image processing is determined to be required. For example, the electronic device can perform inference operations on a partial image corresponding to a second region of interest (522) after the inference operations for a partial image (810) corresponding to a first region of interest (521) have been completed.

[0122] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 5) may process the inference operations described with reference to FIG. 7 and FIG. 8 later than the inference operations described with reference to FIG. 6. The electronic device (e.g., the electronic device (101) of FIG. 5) may perform operations sequentially according to the order in which the inference operations described with reference to FIG. 7 and the inference operations described with reference to FIG. 8 are requested from the neural network processing device (700, 800).

[0123] FIG. 9 is a drawing for illustrating an embodiment in which an electronic device (e.g., the electronic device (101) of FIG. 1) according to one embodiment composites an improved region of interest to an image. FIG. 9 is a drawing for illustrating an embodiment in which a second partial image (820) described with reference to FIG. 8 is composited to a second image (730) described with reference to FIG. 7. The second image (930) of FIG. 9 may correspond to the second image (730) described with reference to FIG. 7. The second partial image (901) of FIG. 9 may correspond to the second partial image (820) described with reference to FIG. 8. The region of interest (940) of FIG. 9 may correspond to the region of interest (520) described with reference to FIG. 5.

[0124] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 5) can identify a second partial image (901). For example, the electronic device can identify a second partial image (910, 920) stored in a memory (e.g., the memory (130) of FIG. 1).

[0125] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 5) can identify the location where the second partial image (901) is to be composited onto the second image (930). For example, the electronic device can identify the location where the second partial image (910, 920) is to be composited based on coordinates included in objectification data (e.g., the objectification data (690) of FIG. 6). For example, the electronic device can place the second partial image (910, 920) corresponding to the region of interest (941, 942) on the second image (930) based on the x-axis coordinate values ​​(e.g., Xstart, Xend) and y-axis coordinate values ​​(e.g., Ystart, Yend) of the region of interest (941, 942). For example, the electronic device may place a second partial image (910) at a location corresponding to a first region of interest (521) and place a second partial image (920) at a location corresponding to a second region of interest (522).

[0126] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 5) can obtain a third image by synthesizing a second partial image (901) to a second image (930). For example, an electronic device (e.g., the electronic device (101) of FIG. 5) can obtain a third image by synthesizing all of the second partial images (910, 920) to which image processing has been applied to the second image (930).

[0127] According to one embodiment, an electronic device can synthesize the second partial image (910, 920) into the second image (930) using matrices (911, 912) corresponding to the second partial image (910, 920). For example, the matrices (911, 912) may be included in objectification data (e.g., objectification data (690) of FIG. 6). For example, the matrices (911, 912) may include probability values ​​corresponding to objects of interest. For example, the electronic device can alpha blend the second partial image (910, 920) into the second image (930) using the matrices (911, 912) as a blur array. For example, the electronic device can alpha blend the second partial image (910) into the second image (930) using the probability values ​​included in the matrix (911) as the transparency of the second partial image (910). For example, the electronic device can alpha blend the second partial image (920) to the second image (930) using the probability value included in the matrix (912) as the transparency of the second partial image (920).

[0128] FIG. 10 is a flowchart relating to a method of operation of an electronic device according to one embodiment. FIG. 10 may correspond to one embodiment of the operation of an electronic device (e.g., the electronic device (101) of FIG. 1) described with reference to FIG. 1 to 9. The operation of the electronic device (e.g., the electronic device (101) of FIG. 1) illustrated in FIG. 10 may be performed by at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the processor (320) of FIG. 3, the application processor (400) of FIG. 4) performing operations or controlling components of the electronic device (e.g., the electronic device (101) of FIG. 1).

[0129] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0130] Referring to operation 1010, an electronic device according to one embodiment can acquire a first image (e.g., the first image (510) of FIG. 5) using a camera module (e.g., the camera module (180) of FIG. 1). For example, the electronic device (101) can acquire the first image (510) by controlling the camera module (e.g., the camera module (180) of FIG. 1) to take a picture by executing a camera application. Since operation 1010 can be applied by analogy to the operation of the electronic device (e.g., the electronic device (101) of FIG. 5) described with reference to FIG. 5, redundant content is omitted.

[0131] Referring to operation 1020, an electronic device according to one embodiment can identify a region of interest (e.g., a region of interest (520) of FIG. 5) from a first image (e.g., the first image (510) of FIG. 5). For example, the electronic device can obtain information regarding the location of the region of interest (e.g., coordinates) that is output by applying the first image to an artificial intelligence neural network model (e.g., a first artificial intelligence neural network model (501)). For example, the electronic device can dynamically allocate a memory of the electronic device (e.g., a memory (130) of FIG. 1) for processing the first image (e.g., the first image (510) of FIG. 5) based on the number of regions of interest (e.g., a region of interest (520) of FIG. 5) identified from the first image (e.g., the first image (510) of FIG. 5). Since operation 1020 can be applied by analogy to the operation of the electronic device (e.g., the electronic device (101) of FIG. 5) described with reference to FIG. 5, redundant content is omitted.

[0132] Referring to operation 1030, an electronic device according to one embodiment can identify an object of interest (e.g., object of interest (602) of FIG. 6) and acquire a first partial image (e.g., objectified data (690)) corresponding to the object of interest.

[0133] According to one embodiment, the electronic device may determine whether to perform image processing on a region of interest (e.g., region of interest (601) of FIG. 6). For example, the electronic device may determine whether to perform image processing on a region of interest (e.g., region of interest (601) of FIG. 6) by analyzing the region of interest (e.g., region of interest (601) of FIG. 6) using an artificial intelligence neural network model (e.g., second artificial intelligence neural network model (611) of FIG. 6). For example, the electronic device may determine whether to perform image processing on an area of ​​interest (e.g., area of ​​interest (601) of FIG. 6) containing an object of interest (e.g., object of interest (602) of FIG. 6) based on at least one of the brightness of the object of interest (e.g., object of interest (602) of FIG. 6), noise included in an area corresponding to the object of interest (e.g., object of interest (602) of FIG. 6), or similarity between a first feature (e.g., aesthetic element) identified from the object of interest (e.g., object of interest (602) of FIG. 6) and a set second feature (e.g., defined aesthetic element). For example, the electronic device may identify that at least one of image processing to improve sharpness, image processing to adjust brightness, image processing to remove noise, and image processing to improve aesthetic elements according to a defined setting is required for the area of ​​interest (e.g., area of ​​interest (601) of FIG. 6).

[0134] According to one embodiment, an electronic device can parse an object of interest (e.g., an object of interest (602) of FIG. 6) from a region of interest (e.g., a region of interest (601) of FIG. 6). For example, the electronic device can parse an object of interest (e.g., an object of interest (602) of FIG. 6) included in a region of interest (e.g., a region of interest (601) of FIG. 6) for which image processing is required. For example, the electronic device can parse an object of interest (e.g., an object of interest (602) of FIG. 6) from a region of interest (e.g., a region of interest (601) of FIG. 6) using an artificial intelligence neural network model (e.g., a third artificial intelligence neural network model (621) of FIG. 6). For example, the electronic device can obtain a matrix (e.g., a matrix (692) of FIG. 6) output from an artificial intelligence neural network model (e.g., a third artificial intelligence neural network model (621) of FIG. 6). For example, a matrix (e.g., matrix (692) of FIG. 6) may contain probability values ​​that each of the parts of the region of interest (e.g., region of interest (601) of FIG. 6) corresponds to an object of interest (e.g., object of interest (602) of FIG. 6). For example, the matrix (e.g., matrix (692) of FIG. 6) may be used as a blur array used for alpha blending. For example, the blur array may represent a probability distribution for the parts corresponding to the object of interest (602).

[0135] According to one embodiment, an electronic device may acquire objectification data (e.g., objectification data (690) of FIG. 6) regarding a region of interest (e.g., region of interest (601) of FIG. 6). For example, the objectification data (e.g., objectification data (690) of FIG. 6) may include the coordinates of a region of interest (e.g., region of interest (521, 522) of FIG. 5) within a first image (e.g., first image (510) of FIG. 5). For example, the objectification data (e.g., objectification data (690) of FIG. 6) may include information regarding the size of a region of interest (e.g., region of interest (521, 522) of FIG. 5) within a first image (e.g., first image (510) of FIG. 5). For example, the objectification data (e.g., the objectification data (690) of FIG. 6) may include image data (691) corresponding to a region of interest (e.g., the region of interest (521, 522) of FIG. 5). For example, the objectification data (e.g., the objectification data (690) of FIG. 6) may include a matrix (e.g., the matrix (692) of FIG. 6) for a region of interest (e.g., the region of interest (601) of FIG. 6) that includes probability values ​​in which each part of the region of interest (e.g., the region of interest (601) of FIG. 6) corresponds to an object of interest (e.g., the object of interest (602) of FIG. 6). For example, the objectification data (e.g., the objectification data (690) of FIG. 6) may include shooting information (e.g., aperture value, shutter speed, sensitivity value, color temperature value) of the camera module (e.g., the camera module (180) of FIG. 1) when the first image (e.g., the first image (510) of FIG. 5) is captured. For example, the objectification data (e.g., the objectification data (690) of FIG. 6) may include information regarding the size of the buffer data for performing image processing on the region of interest (e.g., the region of interest (521, 522) of FIG. 5).For example, objectified data (e.g., objectified data (690) of FIG. 6) can be stored in memory (e.g., memory (130) of FIG. 1).

[0136] Since operation 1030 can be applied by analogy to the operation of the electronic device described with reference to FIG. 6, redundant content is omitted.

[0137] Referring to operation 1040, an electronic device according to one embodiment may apply a first image (e.g., the first image (510) of FIG. 5) to a first artificial intelligence neural network model (e.g., the fourth artificial intelligence neural network model (701) of FIG. 7) and apply a first partial image (e.g., the first partial image (810) of FIG. 8) to a second artificial intelligence neural network model (e.g., the fifth artificial intelligence neural network model (801) of FIG. 8). For example, the electronic device may perform inference operations on the first artificial intelligence neural network model (e.g., the fourth artificial intelligence neural network model (701) of FIG. 7) using a neural network processing device (e.g., the neural network processing device (410) of FIG. 4). For example, the electronic device may perform inference operations on the second artificial intelligence neural network model (e.g., the fifth artificial intelligence neural network model (801) of FIG. 8) using a neural network processing device (e.g., the neural network processing device (410) of FIG. 4).

[0138] According to one embodiment, an electronic device may apply a first image (e.g., the first image (510) of FIG. 5) to a first artificial intelligence neural network model (e.g., the fourth artificial intelligence neural network model (701) of FIG. 7). For example, the electronic device may divide the first image (e.g., the first image (510) of FIG. 5) into a plurality of parts (e.g., the plurality of parts (710) of FIG. 7) to fit the batch size of the first artificial intelligence neural network model (e.g., the fourth artificial intelligence neural network model (701) of FIG. 7). For example, the electronic device may sequentially apply the plurality of parts (e.g., the plurality of parts (710) of FIG. 7) of the first image (e.g., the first image (510) of FIG. 5) to the first artificial intelligence neural network model (e.g., the fourth artificial intelligence neural network model (701) of FIG. 7). For example, the first artificial intelligence neural network model (e.g., the fourth artificial intelligence neural network model (701) of FIG. 7) may be trained to sequentially perform image processing on multiple parts (e.g., multiple parts (710) of FIG. 7) of the input first image (e.g., the first image (510) of FIG. 5).

[0139] According to one embodiment, an electronic device may apply a first partial image (e.g., the first partial image (810) of FIG. 8) to a second artificial intelligence neural network model (e.g., the fifth artificial intelligence neural network model (801) of FIG. 8). For example, the electronic device may resize the first partial image (e.g., the first partial image (810) of FIG. 8) to fit the batch size of the second artificial intelligence neural network model (e.g., the fifth artificial intelligence neural network model (801) of FIG. 8).

[0140] For example, a second artificial intelligence neural network model (e.g., the fifth artificial intelligence neural network model (801) of FIG. 8) may be trained to perform image processing such that a first feature (e.g., an aesthetic element) identified from an object of interest (e.g., an object of interest (602) of FIG. 6) is similar to a second feature (e.g., a defined aesthetic element).

[0141] Since operation 1040 can be applied by analogy to the operation of the electronic device described with reference to FIGS. 7 and FIGS. 8, redundant content is omitted.

[0142] Referring to operation 1050, an electronic device according to one embodiment can acquire a second image (e.g., the second image (730) of FIG. 7) and a second partial image (e.g., the second partial image (820) of FIG. 8) based on a task sequence input to a neural network processing device (e.g., the neural network processing device (410) of FIG. 4).

[0143] According to one embodiment, an electronic device can obtain a second image (e.g., a second image (730) of FIG. 7) that is output by applying a first image (e.g., a first image (510) of FIG. 5) to a first artificial intelligence neural network model (e.g., a fourth artificial intelligence neural network model (701) of FIG. 7). For example, the electronic device can obtain a second image (e.g., a second image (730) of FIG. 7) in which at least one image processing is applied to the first image (e.g., a first image (510) of FIG. 5) through the first artificial intelligence neural network model (e.g., a fourth artificial intelligence neural network model (701) of FIG. 7). For example, an electronic device may obtain a second image (e.g., the second image (730) of FIG. 7) to which at least one image processing is applied, among image processing that removes noise included in the first image (e.g., the first image (510) of FIG. 5), image processing that improves the clarity of the first image (e.g., the first image (510) of FIG. 5), image processing that adjusts the color of the first image (e.g., the first image (510) of FIG. 5), image processing that adjusts the brightness of the first image (e.g., the first image (510) of FIG. 5), image processing that adjusts the contrast of the first image (e.g., the first image (510) of FIG. 5), image processing that improves the detail of the first image (e.g., the first image (510) of FIG. 5), or image processing that identifies and removes unnecessary objects from the first image (e.g., the first image (510) of FIG. 5).

[0144] According to one embodiment, an electronic device can obtain a second partial image (e.g., a second partial image (820) of FIG. 8) output by applying a first partial image (e.g., a first partial image (810) of FIG. 8) to a second artificial intelligence neural network model (e.g., a fifth artificial intelligence neural network model (801) of FIG. 8). For example, the electronic device can obtain a second partial image (e.g., a second partial image (820) of FIG. 8) in which at least one image processing is applied to the first partial image (e.g., a first partial image (810) of FIG. 8) through the second artificial intelligence neural network model (e.g., a fifth artificial intelligence neural network model (801) of FIG. 8). For example, the electronic device may apply at least one image processing to a first partial image (810) containing an object of interest (e.g., object of interest (602) in FIG. 6) based on a decision to perform image processing on an object of interest (e.g., object of interest (602) in FIG. 6).For example, the electronic device may obtain a second partial image (e.g., the second partial image (820) of FIG. 8) to which at least one image processing is applied, among image processing to remove noise included in the first partial image (810), image processing to improve the sharpness of the first partial image (810), image processing to adjust the color of the first partial image (810), image processing to adjust the brightness of the first partial image (810), image processing to adjust the contrast of the first partial image (810), image processing to improve the detail of the first partial image (810), image processing to identify and remove unnecessary objects from the first partial image (810), image processing to modify aesthetic elements included in the first partial image (810) (e.g., wrinkles of a person, blemishes on the skin, makeup) to meet defined standards, or image processing to adjust the sharpness of a high-frequency region (e.g., hair) of the first partial image (810).

[0145] Since the operation 1050 can be applied by analogy to the operation of the electronic device described with reference to FIGS. 7 and FIGS. 8, redundant content is omitted.

[0146] Referring to operation 1060, an electronic device according to one embodiment can obtain a third image by compositing a second partial image (e.g., the second partial image (901) of FIG. 9) with a second image (e.g., the second image (930) of FIG. 9). For example, the electronic device can identify the second partial image (e.g., the second partial image (901) of FIG. 9). For example, the electronic device can identify the location where the second partial image (e.g., the second partial image (901) of FIG. 9) is to be composited onto the second image (e.g., the second image (930) of FIG. 9). For example, the electronic device can place the second partial image (e.g., the second partial image (901) of FIG. 9) at the identified location and composite it with the second image (e.g., the second image (930) of FIG. 9).

[0147] According to one embodiment, an electronic device can synthesize a second partial image (e.g., the second partial image (901) of FIG. 9) onto a second image (e.g., the second image (930) of FIG. 9) using a matrix containing probability values ​​corresponding to an object of interest (e.g., the matrix (911, 912)) of FIG. 9. For example, the matrix (911, 912) may be included in objectification data (e.g., the objectification data (690) of FIG. 6). For example, the electronic device can alpha blend the second partial image (910, 920) onto the second image (930) using the matrix (911, 912) as a blur array. For example, the electronic device can alpha blend the second partial image (910) onto the second image (930) using the probability values ​​included in the matrix (911) as the transparency of the second partial image (910).

[0148] Since operation 1060 can be applied by analogy to the operation of the electronic device described with reference to FIG. 9, redundant content is omitted.

[0149] According to one embodiment, when the process is interrupted while the electronic device is performing operations 1010 to 1060, it may only perform the operation of obtaining a second image (e.g., the second image (730) of FIG. 7) by applying a first image (e.g., the first image (510) of FIG. 5) to a first artificial intelligence neural network model (e.g., the fourth artificial intelligence neural network model (701) of FIG. 7). For example, the electronic device may interrupt the process when it is identified that the remaining charge of the electronic device's battery is below a reference level. For example, the electronic device may interrupt the process when a sudden termination of the camera application occurs.

[0150] FIG. 11 is a diagram illustrating the flow of operation of an electronic device according to one embodiment. FIG. 11 may correspond to one embodiment of the operation of an electronic device (e.g., the electronic device (101) of FIG. 1) described with reference to FIG. 1 through 9.

[0151] According to one embodiment, an electronic device (101) can acquire a first image (1101). For example, the electronic device (101) can acquire the first image (1101) using a camera module (e.g., the camera module (180) of FIG. 1). For example, the electronic device (101) can acquire the first image (1101) stored in a memory (e.g., the memory (130) of FIG. 1). For example, the electronic device (101) can download the first image (1101).

[0152] According to one embodiment, the electronic device (101) can perform a first artificial intelligence operation (1110) on a first image (1101). For example, the first artificial intelligence operation (1110) may be an operation that processes tasks with a small amount of computation first (SJF, shortest job first). For example, the first artificial intelligence operation (1110) may be performed by a neural network processing unit of the electronic device (101) (for example, the neural network processing unit (NPU) (410) of FIG. 4). For example, the neural network processing unit (for example, the neural network processing unit (NPU) (410) of FIG. 4) may perform operations sequentially according to the order of the requested tasks.

[0153] According to one embodiment, the first artificial intelligence operation (1110) may include a region of interest detection operation (1120) for identifying regions of interest (1121, 1122, 1123, 1124) from the first image (1101). For example, a neural network processing unit (e.g., the neural network processing unit (NPU) (410) of FIG. 4) may perform an operation to identify regions of interest (1121, 1122, 1123, 1124) from the first image (1101) using a first artificial intelligence neural network model (e.g., the first artificial intelligence neural network model (501) of FIG. 5). The operation of the electronic device described with reference to FIG. 5 may be applied by analogy to the region of interest detection operation (1120). Redundant content is omitted.

[0154] According to one embodiment, when the electronic device (101) performs the first artificial intelligence operation (1110), it may allocate memory corresponding to the number of objects of interest (1121, 1122, 1123, 1124) to the inference operation. For example, the electronic device (101) may allocate memory corresponding to the number of regions of interest (1121, 1122, 1123, 1124) detected from the first image (1101) as a result of performing the region of interest detection operation (1120) to the region of interest analysis operation (1130) and the object of interest parsing operation (1140).

[0155] According to one embodiment, the first artificial intelligence operation (1110) may include an interest region analysis operation (1130) that analyzes the interest regions (1121, 1122, 1123, 1124). For example, a neural network processing unit (e.g., the neural network processing unit (NPU) (410) of FIG. 4) may perform an inference operation to identify whether image processing is required for the interest regions (1121, 1122, 1123, 1124) using a second artificial intelligence neural network model (e.g., the second artificial intelligence neural network model (611) of FIG. 6). For example, if there are multiple interest regions (1121, 1122, 1123, 1124), the interest region analysis operation (1130) may be performed on all of the multiple interest regions (1121, 1122, 1123, 1124). For example, the region of interest analysis operation (1130) may include an inference operation that identifies that image processing is required for at least some of the multiple regions of interest (1121, 1122, 1123, 1124) (e.g., 1121, 1122). For example, the region of interest analysis operation (1130) may include an inference operation that identifies that image processing is not required for at least some of the multiple regions of interest (1121, 1122, 1123, 1124) (e.g., 1123, 1124). The operation of the electronic device described with reference to FIG. 6 may be applied by analogy to the region of interest analysis operation (1130). Redundant content is omitted.

[0156] According to one embodiment, the first artificial intelligence operation (1110) may perform an interest region parsing operation (1140) for parsing an object of interest from an interest region (1121, 1122, 1123, 1124). For example, a neural network processing unit (e.g., the neural network processing unit (NPU) (410) of FIG. 4) may perform an inference operation for parsing an object of interest using a third artificial intelligence neural network model (e.g., the third artificial intelligence neural network model (621) of FIG. 6). For example, the interest region parsing operation (1140) may include an inference operation for parsing an object of interest from an interest region (e.g., 1121, 1122) identified as requiring image processing. For example, a neural network processing unit (e.g., the neural network processing unit (NPU) (410) of FIG. 4) can perform an interest region parsing operation (1140) to sequentially parse objects of interest for all interest regions identified as requiring image processing. The operation of the electronic device described with reference to FIG. 6 can be applied by analogy to the interest region parsing operation (1140). Redundant content is omitted.

[0157] According to one embodiment, the electronic device (101) can perform a second artificial intelligence operation (1150) on a first image (1101). For example, the second artificial intelligence operation (1150) may include more complex operations than the first artificial intelligence operation (1110). For example, the second artificial intelligence operation (1150) may be performed by a neural network processing unit of the electronic device (101) (e.g., the neural network processing unit (NPU) (410) of FIG. 4). For example, the neural network processing unit (e.g., the neural network processing unit (NPU) (410) of FIG. 4) may perform operations sequentially according to the order of the requested tasks. For example, a neural network processing unit (e.g., the neural network processing unit (NPU) (410) of FIG. 4) can sequentially perform operations according to the request order of the task for image processing operations (1160) performed using the fourth artificial intelligence neural network model (e.g., the fourth artificial intelligence neural network model (701) of FIG. 7) and the task for region of interest processing operations (1160) performed using the fifth artificial intelligence neural network model (e.g., the fifth artificial intelligence neural network model (801) of FIG. 8).

[0158] According to one embodiment, the second artificial intelligence operation (1150) may include an image processing operation (1160) that improves the first image (1101). For example, a neural network processing unit (e.g., the neural network processing unit (NPU) (410) of FIG. 4) may perform an inference operation that improves the first image (1101) using a fourth artificial intelligence neural network model (e.g., the fourth artificial intelligence neural network model (701) of FIG. 7). For example, the image processing operation (1160) may include an inference operation that removes noise contained in the first image (1101). For example, the image processing operation (1160) may include an inference operation that improves the clarity of the first image (1101). For example, the image processing operation (1160) may include an inference operation that adjusts the color of the first image (1101). For example, the image processing operation (1160) may include an inference operation that adjusts the brightness of the first image (1101). For example, the image processing operation (1160) may include an inference operation that adjusts the contrast of the first image (1101). For example, the image processing operation (1160) may include an inference operation that enhances the detail of the first image (1101). For example, the image processing operation (1160) may include an inference operation that increases the resolution of the first image (1101). For example, the image processing operation (1160) may include an inference operation that identifies and removes unnecessary objects from the first image (1101). For example, the image processing operation (1160) may include an operation that improves the first image (1101) by dividing it into a plurality of parts (e.g., 1101a). For example, the image processing operation (1160) may include an operation to generate a second image (1102) by combining the improved parts. For example, the operation of the electronic device described with reference to FIG. 7 may be applied by analogy to the image processing operation (1160). Duplicate content is omitted.

[0159] According to one embodiment, the second artificial intelligence operation (1150) may include a region of interest processing operation (1170) that improves a region of interest (e.g., 1121, 1122). For example, the region of interest processing operation (1170) may be applied to a region of interest (e.g., 1121, 1122) identified as requiring image processing as a result of performing a region of interest analysis operation (1130). For example, a neural network processing unit (e.g., the neural network processing unit (NPU) (410) of FIG. 4) may perform an inference operation that improves a first partial image (1171) corresponding to a region of interest (e.g., 1121) using a fourth artificial intelligence neural network model (e.g., the fourth artificial intelligence neural network model (701) of FIG. 7). For example, the region of interest processing operation (1170) may include an inference operation that removes noise contained in the first partial image (1171). For example, the region of interest processing operation (1170) may include an inference operation that improves the sharpness of the first partial image (1171). For example, the region of interest processing operation (1170) may include an inference operation that adjusts the color of the first partial image (1171). For example, the region of interest processing operation (1170) may include an inference operation that adjusts the brightness of the first partial image (1171). For example, the region of interest processing operation (1170) may include an inference operation that adjusts the contrast of the first partial image (1171). For example, the region of interest processing operation (1170) may include an inference operation that improves the detail of the first partial image (1171). For example, the region of interest processing operation (1170) may include an inference operation that increases the resolution of the first partial image (1171). For example, the region of interest processing operation (1170) may include an inference operation to identify and remove unnecessary objects from the first partial image (1171).For example, the region of interest processing operation (1170) may include an operation to generate a second partial image (1172) by improving a first partial image (1171). For example, a neural network processing unit (e.g., the neural network processing unit (NPU) (410) of FIG. 4) may sequentially perform the region of interest processing operation (1170) on a plurality of regions of interest (e.g., 1121, 1122). For example, the neural network processing unit (e.g., the neural network processing unit (NPU) (410) of FIG. 4) may generate a second partial image (1173) by performing the region of interest processing operation (1170) on another region of interest (e.g., 1122) after the inference operation for a region of interest (e.g., 1121) is completed. For example, the operation of the electronic device described with reference to FIG. 8 may be applied by analogy to the region of interest processing operation (1170). Duplicate content is omitted.

[0160] According to one embodiment, the electronic device (101) can synthesize an improved region of interest into an improved image. For example, the electronic device (101) can generate a third image (1103) by synthesizing a second partial image (1172, 1173) into a second image (1102). For example, the electronic device can alpha blend the second partial image (1172, 1173) into the second image (1102) by using a matrix (e.g., matrix (911) of FIG. 9) obtained through an object of interest parsing operation (1140) as a blur array. The operation of the electronic device (101) synthesizing the second partial image (1172, 1173) into the second image (1102) can be inferred from the operation of the electronic device described with reference to FIG. 9. Redundant content is omitted.

[0161] FIG. 12 is a drawing showing a system including a generative artificial intelligence model according to one embodiment.

[0162] According to one embodiment, the electronic device of FIGS. 1 to 4 (e.g., 101 of FIG. 1) may be configured to include at least some of the User Query / Response Interface (1210) AI framework (1220), application / service component (1230), knowledge repositories (1240), or Generative AI Model (1250) of FIG. 12. According to one embodiment, at least some of the User Query / Response Interface (1210) AI framework (1220), application / service component (1230), knowledge repositories (1240), or Generative AI Model (1250) of FIG. 12 may be included in an external electronic device (e.g., 102 of FIG. 1).

[0163] Referring to FIG. 12, the User Query / Response Interface (1210) can receive user input. The user input may be in the form of natural language, images, and / or videos. Additionally, context information may be transmitted along with the user input. Context information may include various additional information at the time of user input. For example, there may be information about the application currently being used by the user or the user's location information. Furthermore, user input may be in a mixed form of the aforementioned natural language, images, sounds, and context information. Additionally, user input may be in a non-natural language form, such as selecting a menu. The User Query / Response Interface (1210) can output results from a generative artificial intelligence system to the user. The output may be in the form of natural language or specific content, and may also be provided in the form of an action requested by the user. The User Query / Response Interface (1210) can output results from a generative artificial intelligence system to the user. The output can be in the form of natural language or specific content, and it can also be provided in a form such as the action requested by the user.

[0164] The AI ​​framework (1220) can receive input from the user and coordinate and control each component necessary to perform the user's intent based on the user's query.

[0165] User input received from the User Query / Response Interface (1210) can be sent to a Prompt design component (1221). The Prompt design component (1221) can be used to generate a prompt suitable for inputting user input into a Large Language Model (LLM) or Large Multimodal Models (LMM). The Prompt design component (1221) may be an AI component that uses machine learning algorithms or neural networks to develop better prompts over time. Based on user input, the Prompt design component (1221) can generate a prompt by accessing a knowledge component (e.g., knowledge repositories (1240)) containing user preference data, a prompt library, and prompt examples, and can pass the generated prompt to the LLM or LMM.

[0166] The API / Plug-in management component (1223) can perform the role of communicating with external information when there is a request for additional information when passing user input as input to a generative model. The API / Plug-in management component (1223) establishes a channel to communicate with the outside of the AI ​​Interface via the API, and can enable access to various data sources (e.g., knowledge repositories (1240)) through the established channel. Additionally, if the application or service needs to perform an action that executes the user input as a final step rather than an intermediate result, the API / Plug-in management component (1223) can request that action from the application / service component (1230) via the API. The information obtained from the outside can be used to generate a prompt in the Prompt design component (1221) along with the user input, or it can be passed as input to the generative model.

[0167] The Refiner component (e.g., output modification component (1225)) allows for detailed tuning of the output from a generative model. For instance, the Refiner component can verify whether the content generated by LLM and / or LMM is irrelevant, contains biased content, or includes harmful content. Additionally, the Refiner component can determine the extent to which the output matches the user's desired outcome and, if necessary, proceed with additional processing. Furthermore, the Refiner component can configure and provide hints to the user to help avoid unwanted outputs.

[0168] A Generative AI Model (1250) generally refers to an artificial intelligence neural network that generates new forms of data based on user input information. A Generative AI Model (1250) may include a model that generates images and / or a model that generates language. Models that generate images include, but are not limited to, GANs (generative adversarial networks) and VAEs (variational autoencoders), and examples include Diffusion-based generative models that use VAEs and Transformer structures. Models that generate language are models trained to output the most statistically appropriate output value based on input values, and examples include models such as CHAT-GPT 3 and CHAT-GPT 4. There are also LMMs that can recognize various forms of data input, such as text, images, and voice, and generate new data corresponding to them.

[0169] The disclosed embodiments have been described based on mobile devices, but are not limited thereto. The disclosed embodiments may be applied to various types of electronic devices utilizing artificial intelligence neural network models (e.g., personal computers (PCs), tablet personal computers, video see-through (VST) electronic devices, automobile devices).

[0170] The disclosed embodiments can be used in various software (e.g., applications, solutions) that utilize artificial intelligence neural network models. For example, the disclosed embodiments can be used in augmented reality applications to provide the effect of smooth and rapid screen transitions. For example, the disclosed embodiments can be used in image editing software to reduce processing time by applying filters or making corrections based on regions of interest.

[0171] According to the disclosed embodiment, the problem of time consumption due to overhead caused by the neural network processing device frequently changing the artificial intelligence neural network model can be reduced through efficient resource scheduling.

[0172] A method of operation of an electronic device (e.g., the electronic device (101) of FIG. 1) according to a disclosed embodiment may include an operation of acquiring a first image (e.g., the first image (510) of FIG. 5) using a camera (e.g., the camera module (180) of FIG. 1) of the electronic device (e.g., the electronic device (101) of FIG. 1). A method of operation of an electronic device (e.g., the electronic device (101) of FIG. 1) may include an operation of performing a first artificial intelligence operation (e.g., the first artificial intelligence operation (1110) of FIG. 11) on the first image (e.g., the first image (510) of FIG. 5) using an NPU (e.g., the neural network processing unit (410) of FIG. 4) of the electronic device (e.g., the electronic device (101) of FIG. 1). A method of operation of an electronic device (e.g., the electronic device (101) of FIG. 1) may include an operation of performing a second artificial intelligence operation (e.g., the second artificial intelligence operation (1150) of FIG. 11) using an NPU (e.g., the neural network processing unit (410) of FIG. 4) of the electronic device (e.g., the electronic device (101) of FIG. 1) based on the result of performing a first artificial intelligence operation (e.g., the second artificial intelligence operation (1150) of FIG. 1). A method of operation of an electronic device (e.g., the electronic device (101) of FIG. 1) may include an operation of obtaining a synthetic image for a first image (e.g., the first image (510) of FIG. 5) using an NPU (e.g., the neural network processing unit (410) of FIG. 4) based on the result of performing a second artificial intelligence operation (e.g., the second artificial intelligence operation (1150) of FIG. 11). The operation of performing the first artificial intelligence operation (e.g., the first artificial intelligence operation (1110) of FIG. 11) may include the operation of identifying a region of interest (e.g., the region of interest (520) of FIG. 5) from the first image (e.g., the first image (510) of FIG. 5).The operation of performing the first artificial intelligence operation (e.g., the first artificial intelligence operation (1110) of FIG. 11) may include identifying an object of interest (e.g., the object of interest (602) of FIG. 6) from a region of interest (e.g., the region of interest (520) of FIG. 5) and obtaining a first partial image (e.g., the first partial image (810) of FIG. 8) corresponding to the object of interest (e.g., the object of interest (602) of FIG. 6). The operation of performing a second artificial intelligence operation (e.g., the second artificial intelligence operation (1150) of FIG. 11) may include an operation method of an electronic device (e.g., the electronic device (101) of FIG. 1) that uses an NPU (e.g., the neural network processing unit (410) of FIG. 4) of the electronic device (e.g., the electronic device (101) of FIG. 1) that applies a first image (e.g., the first image (510) of FIG. 5) to a first artificial intelligence neural network model (e.g., the fourth artificial intelligence neural network model (701) of FIG. 7) that uses an NPU (e.g., the neural network processing unit (410) of FIG. 4) and applies a first partial image (e.g., the first partial image (810) of FIG. 8) to a second artificial intelligence neural network model (e.g., the fifth artificial intelligence neural network model (801) of FIG. 8) that uses an NPU (e.g., the neural network processing unit (410) of FIG. 4).The operation of performing a second artificial intelligence operation (e.g., the second artificial intelligence operation (1150) of FIG. 11) is based on the task sequence input to the NPU (e.g., the neural network processing unit (410) of FIG. 4) from the first artificial intelligence neural network model (e.g., the fourth artificial intelligence neural network model (701) of FIG. 7) and the second artificial intelligence neural network model (e.g., the fifth artificial intelligence neural network model (801) of FIG. 8), by using the NPU (e.g., the neural network processing unit (410) of FIG. 4) to perform image processing on the first image (e.g., the first image (510) of FIG. 5)) using the NPU, thereby performing image processing on the second image (e.g., the second image (730) of FIG. 7) and the first partial image (e.g., the first partial image (810) of FIG. 8) output from the first artificial intelligence neural network model (e.g., the fourth artificial intelligence neural network model (701) of FIG. 7), so as to perform image processing on the second image (e.g., the second image (730) of FIG. 7) and the first partial image (e.g., the first partial image (810) of FIG. 8) of FIG. 8) output from the first artificial intelligence neural network The operation of obtaining a second partial image (e.g., the second partial image (820) of FIG. 8) output from a model (e.g., the fifth artificial intelligence neural network model (801) of FIG. 8). The operation of obtaining a composite image may include obtaining a third image by synthesizing the second partial image (e.g., the second partial image (901) of FIG. 9) with a second image (e.g., the second image (930) of FIG. 9).

[0173] According to one embodiment, the operation of identifying a region of interest (e.g., the region of interest (520) of FIG. 5) may include the operation of identifying a region of interest (e.g., the region of interest (520) of FIG. 5) from a first image by applying a first image (e.g., the first image (510) of FIG. 5) to a third artificial intelligence neural network model (e.g., (501) of FIG. 5). The operation of acquiring a first partial image (e.g., the first partial image (810) of FIG. 8) may include the operation of acquiring a blur array (e.g., the matrix (692) of FIG. 6) for a portion corresponding to an object of interest (e.g., the object of interest (602) of FIG. 6) from a region of interest (e.g., the region of interest (520) of FIG. 5). The operation of acquiring the third image may include the operation of compositing the second partial image (e.g., the second partial image (901) of FIG. 9) with the second image (e.g., the second image (930) of FIG. 9) using a blur array (e.g., the matrix (911, 912) of FIG. 9).

[0174] According to one embodiment, the operation of obtaining a blur array (e.g., the matrix (692) of FIG. 6) may include applying a first image (e.g., the first image (510) of FIG. 5) to a fourth artificial intelligence neural network model (e.g., the third artificial intelligence neural network model (621) of FIG. 6)) and obtaining the blur array (e.g., the matrix (692) of FIG. 6) based on a probability distribution for a portion corresponding to an object of interest (e.g., the object of interest (602) of FIG. 6) output from the fourth artificial intelligence neural network model (e.g., the third artificial intelligence neural network model (621) of FIG. 6).

[0175] According to one embodiment, the operation of acquiring a first partial image (e.g., the first partial image (810) of FIG. 8) may include an operation of acquiring objectification data comprising at least one of information regarding the coordinates of a region of interest (e.g., the region of interest (520) of FIG. 5), the size of the region of interest, information regarding a portion corresponding to an object of interest (e.g., the object of interest (602) of FIG. 6), and information regarding a blur array (e.g., the matrix (692) of FIG. 6).

[0176] According to one embodiment, the operation of acquiring a first partial image (e.g., the first partial image (810) of FIG. 8) may include an operation of determining whether to perform image processing on a region of interest (e.g., the region of interest (601) of FIG. 6) that includes an object of interest (e.g., the object of interest (602) of FIG. 6). The operation of applying the first partial image (e.g., the first partial image (810) of FIG. 8) to a second artificial intelligence neural network model (e.g., the fifth artificial intelligence neural network model (801) of FIG. 8) may include an operation of applying the first partial image (e.g., the first partial image (810) of FIG. 8) to the second artificial intelligence neural network model (e.g., the fifth artificial intelligence neural network model (801) of FIG. 8) based on a decision to perform image processing on a region of interest (e.g., the region of interest (601) of FIG. 6).

[0177] According to one embodiment, the operation of determining whether to perform image processing on a region of interest (e.g., region of interest (601) of FIG. 6) may include the operation of determining whether to perform image processing on a region of interest (e.g., region of interest (601) of FIG. 6) based on at least one of the brightness of an object of interest (e.g., object of interest (602) of FIG. 6), noise included in a region corresponding to an object of interest (e.g., object of interest (602) of FIG. 6), or the similarity between a first feature identified from an object of interest (e.g., object of interest (602) of FIG. 6) and a set second feature.

[0178] According to one embodiment, a method of operating an electronic device (e.g., the electronic device (101) of FIG. 1) may include an operation of dynamically allocating memory (e.g., (130) of FIG. 1) of the electronic device (e.g., the electronic device (101) of FIG. 1) for processing the first image (e.g., the first image (510) of FIG. 5) based on the number of regions of interest (e.g., the region of interest (520) of FIG. 5) identified from the first image (e.g., the first image (510) of FIG. 5).

[0179] According to one embodiment, the operation of applying a first image (e.g., the first image (510) of FIG. 5) to a first artificial intelligence neural network model (e.g., the fourth artificial intelligence neural network model (701) of FIG. 7) may include the operation of dividing the first image (e.g., the first image (510) of FIG. 5) into a plurality of parts (e.g., the plurality of parts (710) of FIG. 7) to fit the batch size of the first artificial intelligence neural network model (e.g., the fourth artificial intelligence neural network model (701) of FIG. 7). The operation of applying a first image (e.g., the first image (510) of FIG. 5) to a first artificial intelligence neural network model (e.g., the fourth artificial intelligence neural network model (701) of FIG. 7) may include the operation of sequentially applying parts of the first image (e.g., the first image (510) of FIG. 5) (e.g., a plurality of parts (710) of FIG. 7) to the first artificial intelligence neural network model (e.g., the fourth artificial intelligence neural network model (701) of FIG. 7). The first artificial intelligence neural network model (e.g., the fourth artificial intelligence neural network model (701) of FIG. 7) may be trained to sequentially perform image processing on parts of the input first image (e.g., the first image (510) of FIG. 5) (e.g., a plurality of parts (710) of FIG. 7).

[0180] According to one embodiment, the operation of applying a first partial image (e.g., the first partial image (810) of FIG. 8) to a second artificial intelligence neural network model (e.g., the fifth artificial intelligence neural network model (801) of FIG. 8) may include the operation of resizing the first partial image (e.g., the first partial image (810) of FIG. 8) to fit the batch size of the second artificial intelligence neural network model (e.g., the fifth artificial intelligence neural network model (801) of FIG. 8). The second artificial intelligence neural network model (e.g., the fifth artificial intelligence neural network model (801) of FIG. 8) may be trained to perform image processing such that a first feature identified from an object of interest (e.g., the object of interest (602) of FIG. 6) is similar to a set second feature.

[0181] A computer-readable, non-transient recording medium having instructions for controlling an electronic device according to one disclosed embodiment may include instructions for acquiring a first image (e.g., the first image (510) of FIG. 5) using a camera (e.g., the camera module (180) of FIG. 1) of an electronic device (e.g., the electronic device (101) of FIG. 1). The recording medium may include instructions for performing a first artificial intelligence operation (e.g., the first artificial intelligence operation (1110) of FIG. 11) on the first image (e.g., the first image (510) of FIG. 5) using an NPU (e.g., the neural network processing unit (410) of FIG. 4) of an electronic device (e.g., the electronic device (101) of FIG. 1). The recording medium may include instructions for performing a second artificial intelligence operation (e.g., the second artificial intelligence operation (1150) of FIG. 11) using an NPU (e.g., the neural network processing unit (410) of FIG. 4) of an electronic device (e.g., the electronic device (101) of FIG. 1) based on the result of performing a first artificial intelligence operation (e.g., the second artificial intelligence operation (1150) of FIG. 11)) using an NPU (e.g., the neural network processing unit (410) of FIG. 4) based on the result of performing a second artificial intelligence operation (e.g., the second artificial intelligence operation (1150) of FIG. 11). The recording medium may include instructions for obtaining a synthetic image for a first image (e.g., the first image (510) of FIG. 5) using an NPU (e.g., the neural network processing unit (410) of FIG. 4) based on the result of performing a second artificial intelligence operation (e.g., the second artificial intelligence operation (1150) of FIG. 11). A command for performing a first artificial intelligence operation (e.g., the first artificial intelligence operation (1110) of FIG. 11) may include a command for identifying a region of interest (e.g., the region of interest (520) of FIG. 5) from a first image (e.g., the first image (510) of FIG. 5).A command for performing a first artificial intelligence operation (e.g., the first artificial intelligence operation (1110) of FIG. 11) may include a command for identifying an object of interest (e.g., the object of interest (602) of FIG. 6) from a region of interest (e.g., the region of interest (520) of FIG. 5) and acquiring a first partial image (e.g., the first partial image (810) of FIG. 8) corresponding to the object of interest (e.g., the object of interest (602) of FIG. 6). The instruction for performing the second artificial intelligence operation (e.g., the second artificial intelligence operation (1150) of FIG. 11) may include an instruction for applying a first image (e.g., the first image (510) of FIG. 5) to a first artificial intelligence neural network model (e.g., the fourth artificial intelligence neural network model (701) of FIG. 7) using an NPU (e.g., the neural network processing unit (410) of FIG. 4) of an electronic device (e.g., the electronic device (101) of FIG. 1), and applying a first partial image (e.g., the first partial image (810) of FIG. 8) to a second artificial intelligence neural network model (e.g., the fifth artificial intelligence neural network model (801) of FIG. 8) using an NPU (e.g., the neural network processing unit (410) of FIG. 4).The instruction for performing the second artificial intelligence operation (e.g., the second artificial intelligence operation (1150) of FIG. 11) is based on the task sequence input to the NPU (e.g., the neural network processing unit (410) of FIG. 4) from the first artificial intelligence neural network model (e.g., the fourth artificial intelligence neural network model (701) of FIG. 7) and the second artificial intelligence neural network model (e.g., the fifth artificial intelligence neural network model (801) of FIG. 8), and image processing is performed on the first image (e.g., the first image (510) of FIG. 5) using the NPU (e.g., the neural network processing unit (410) of FIG. 4), thereby performing image processing on the second image (e.g., the second image (730) of FIG. 7) and the first partial image (e.g., the first partial image (810) of FIG. 8) output from the first artificial intelligence neural network model (e.g., the fourth artificial intelligence neural network model (701) of FIG. 7), so as to perform image processing on the second image (e.g., the second image (730) of FIG. 7) and the first partial image (e.g., the first partial image (810) of FIG. 8) of FIG. 8, thereby the second artificial intelligence neural network The command for obtaining a second partial image (e.g., the second partial image (820) of FIG. 8) output from a model (e.g., the fifth artificial intelligence neural network model (801) of FIG. 8). The command for obtaining a composite image may include a command for obtaining a third image by synthesizing the second partial image (e.g., the second partial image (901) of FIG. 9) with a second image (e.g., the second image (930) of FIG. 9).

[0182] An electronic device according to one disclosed embodiment may include at least one processor (e.g., (120) of FIG. 1) including a camera (e.g., camera module (180) of FIG. 1), an NPU (e.g., neural network processing unit (410) of FIG. 4), and a memory (e.g., memory (130) of FIG. 1) for storing instructions. The electronic device may acquire a first image (e.g., first image (510) of FIG. 5) using the camera (e.g., camera module (180) of FIG. 1). The electronic device may perform a first artificial intelligence operation (e.g., first artificial intelligence operation (1110) of FIG. 11) on the first image (e.g., first image (510) of FIG. 5) using the NPU (e.g., neural network processing unit (410) of FIG. 4). Based on the result of performing a first artificial intelligence operation (e.g., the first artificial intelligence operation (1110) of FIG. 11), the electronic device can perform a second artificial intelligence operation (e.g., the second artificial intelligence operation (1150) of FIG. 11) using an NPU (e.g., the neural network processing unit (410) of FIG. 4) of the electronic device (e.g., the electronic device (101) of FIG. 1). Based on the result of performing the second artificial intelligence operation (e.g., the second artificial intelligence operation (1150) of FIG. 11), the electronic device can obtain a synthetic image for a first image (e.g., the first image (510) of FIG. 5) using an NPU (e.g., the neural network processing unit (410) of FIG. 4). The electronic device can identify a region of interest (e.g., a region of interest (520) of FIG. 5) from a first image (e.g., a first image (510) of FIG. 5) as a first artificial intelligence operation (e.g., a first artificial intelligence operation (1110) of FIG. 11). The electronic device can identify an object of interest (e.g., an object of interest (602) of FIG. 6) from a region of interest (e.g., a region of interest (520) of FIG. 5) as a first artificial intelligence operation (e.g., a first artificial intelligence operation (1110) of FIG. 11).The electronic device may acquire a first partial image (e.g., the first partial image (810) of FIG. 8) corresponding to an object of interest (e.g., the object of interest (602) of FIG. 6) as a first artificial intelligence operation (e.g., the first artificial intelligence operation (1110) of FIG. 11). The electronic device may apply the first image (e.g., the first image (510) of FIG. 5) to a first artificial intelligence neural network model (e.g., the fourth artificial intelligence neural network model (701) of FIG. 7) using an NPU (e.g., the neural network processing unit (410) of FIG. 4) as a second artificial intelligence operation (e.g., the second artificial intelligence operation (1150) of FIG. 11). The electronic device can apply a first partial image (e.g., the first partial image (810) of FIG. 8) to a second artificial intelligence neural network model (e.g., the fifth artificial intelligence neural network model (801) of FIG. 8) that uses an NPU (e.g., the second artificial intelligence operation (1150) of FIG. 11) as a second artificial intelligence operation (e.g., the second artificial intelligence operation (1150) of FIG. 11)).The electronic device performs image processing on a first image (e.g., the first image (510) of FIG. 4) using the NPU (e.g., the second AI operation (1150) of FIG. 11) based on a task sequence input to the NPU (e.g., the neural network processing unit (410) of FIG. 4) from a first AI neural network model (e.g., the fourth AI neural network model (701) of FIG. 7) and a second AI neural network model (e.g., the fifth AI neural network model (801) of FIG. 8), thereby performing image processing on a second image (e.g., the first image (510) of FIG. 5) using the NPU (e.g., the neural network processing unit (410) of FIG. 4)) and performing image processing on a second image (e.g., the second image (730) of FIG. 7) and a first partial image (e.g., the first partial image (810) of FIG. 8) output from the first AI neural network model (e.g., the fourth AI neural network model (701) of FIG. 7), so as to perform the second AI neural network A second partial image (e.g., the second partial image (820) of FIG. 8) output from a model (e.g., the fifth artificial intelligence neural network model (801) of FIG. 8) can be obtained. The electronic device can obtain a third image by synthesizing the second partial image (e.g., the second partial image (901) of FIG. 9) with a second image (e.g., the second image (930) of FIG. 9).

[0183] According to one embodiment, the electronic device can identify a region of interest (e.g., a region of interest (520) in FIG. 5) from a first image by applying the first image (e.g., the first image (510) in FIG. 5) to a third artificial intelligence neural network model (e.g., (501) in FIG. 5). The electronic device can obtain a blur array (e.g., a matrix (692) in FIG. 6) for a portion corresponding to an object of interest (e.g., an object of interest (602) in FIG. 6) from the region of interest (e.g., a region of interest (520) in FIG. 5). The electronic device can synthesize a second partial image (e.g., a second partial image (901) in FIG. 9) with a second image (e.g., a second image (930) in FIG. 9) using the blur array (e.g., matrices (911, 912) in FIG. 9).

[0184] According to one embodiment, the electronic device can obtain a blur array (e.g., matrix (692) of FIG. 6) based on a probability distribution for a portion corresponding to an object of interest (e.g., object of interest (602) of FIG. 6) output from the fourth artificial intelligence neural network model (e.g., third artificial intelligence neural network model (621) of FIG. 6) by applying a first image (e.g., the first image (510) of FIG. 5) to a fourth artificial intelligence neural network model (e.g., the third artificial intelligence neural network model (621) of FIG. 6).

[0185] According to one embodiment, the electronic device can obtain objectification data including at least one of information regarding the coordinates of a region of interest (e.g., region of interest (520) in FIG. 5), the size of the region of interest, information regarding a part corresponding to an object of interest (e.g., object of interest (602) in FIG. 6), and information regarding a blur array (e.g., matrix (692) in FIG. 6).

[0186] According to one embodiment, the electronic device may determine whether to perform image processing on a region of interest (e.g., region of interest (601) of FIG. 6) that includes an object of interest (e.g., object of interest (602) of FIG. 6). Based on the determination to perform image processing on the region of interest (e.g., region of interest (601) of FIG. 6), the electronic device may apply a first partial image (e.g., first partial image (810) of FIG. 8) to a second artificial intelligence neural network model (e.g., fifth artificial intelligence neural network model (801) of FIG. 8).

[0187] According to one embodiment, the electronic device may determine whether to perform image processing on a region of interest (e.g., region of interest (601) in FIG. 6) based on at least one of the brightness of an object of interest (e.g., object of interest (602) in FIG. 6), noise included in a region corresponding to the object of interest (e.g., object of interest (602) in FIG. 6), or similarity between a first feature identified from the object of interest (e.g., object of interest (602) in FIG. 6) and a set second feature.

[0188] According to one embodiment, the electronic device may dynamically allocate memory (e.g., (130) of FIG. 1) of the electronic device (e.g., the electronic device (101) of FIG. 1) for processing the first image (e.g., the first image (510) of FIG. 5) based on the number of regions of interest (e.g., the region of interest (520) of FIG. 5) identified from the first image (e.g., the first image (510) of FIG. 5).

[0189] According to one embodiment, the electronic device may divide a first image (e.g., the first image (510) of FIG. 5) into a plurality of parts (e.g., the plurality of parts (710) of FIG. 7) to fit the batch size of a first artificial intelligence neural network model (e.g., the fourth artificial intelligence neural network model (701) of FIG. 7). The electronic device may sequentially apply the parts of the first image (e.g., the first image (510) of FIG. 5) (e.g., the plurality of parts (710) of FIG. 7) to the first artificial intelligence neural network model (e.g., the fourth artificial intelligence neural network model (701) of FIG. 7). The first artificial intelligence neural network model (e.g., the fourth artificial intelligence neural network model (701) of FIG. 7) may be trained to sequentially perform image processing on parts (e.g., multiple parts (710) of FIG. 7) of the input first image (e.g., the first image (510) of FIG. 5).

[0190] According to one embodiment, the electronic device may resize a first partial image (e.g., the first partial image (810) of FIG. 8) to fit the batch size of a second artificial intelligence neural network model (e.g., the fifth artificial intelligence neural network model (801) of FIG. 8). The second artificial intelligence neural network model (e.g., the fifth artificial intelligence neural network model (801) of FIG. 8) may be trained to perform image processing such that a first feature identified from an object of interest (e.g., the object of interest (602) of FIG. 6) is similar to a set second feature.

[0191] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs from the description below.

[0192] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0193] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0194] As used in this document, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be a component formed as a whole, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0195] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' merely means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0196] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0197] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the components of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to the integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In the method of operating the electronic device (101), The operation of acquiring a first image (510) using the camera (180) of the electronic device (101); An operation of performing a first artificial intelligence operation (1110) on the first image (510) using the NPU (410) of the electronic device (101); Based on the result of performing the first artificial intelligence operation (1110), the operation of performing a second artificial intelligence operation (1150) using the NPU (410) of the electronic device (101); and Based on the result of performing the second artificial intelligence operation (1150) above, the operation of obtaining a synthetic image for the first image (510) using the NPU (410) is included. The operation of performing the above first artificial intelligence operation (1110) is, An operation to identify a region of interest (520) from the first image (510); and The method includes identifying an object of interest (602) from the area of ​​interest (520) and obtaining a first partial image (690) corresponding to the object of interest (602). The operation of performing the above second artificial intelligence operation (1150) is, The operation of applying the first image (510) to a first artificial intelligence neural network model (701) using the NPU (410), and applying the first partial image (810) to a second artificial intelligence neural network model (801) using the NPU (410); and Based on the task sequence input to the NPU (410) from the first artificial intelligence neural network model (701) and the second artificial intelligence neural network model (801), image processing is performed on the first image (510) using the NPU (410) to obtain the second image (730) output from the first artificial intelligence neural network model (701) and image processing is performed on the first partial image (810) to obtain the second partial image (820) output from the second artificial intelligence neural network model (801); The operation of acquiring the above-mentioned composite image is, The operation of obtaining a third image by synthesizing the second partial image (901) with the second image (930), Method of operation.

2. In Paragraph 1, The operation of identifying the above region of interest (520) is The method includes the operation of identifying the region of interest (520) from the first image by applying the first image (510) to the third artificial intelligence neural network model (501), and The operation of acquiring the first partial image (690) above is, The method includes the operation of obtaining a blur array (692) for a portion corresponding to the object of interest (602) from the region of interest (520), and The operation of acquiring the above third image is, A method comprising compositing the second partial image (901) with the second image (930) using the blur arrays (911, 912). Method of operation.

3. In Paragraph 2, The operation of obtaining the above blur array (692) is The operation of obtaining the blur array (692) based on the probability distribution of the portion corresponding to the object of interest (602) output from the fourth artificial intelligence neural network model (621) by applying the first image (510) to the fourth artificial intelligence neural network model (621), Method of operation.

4. In Paragraph 2, The operation of acquiring the first partial image (690) above is, The operation of acquiring objectification data including at least one of information regarding the coordinates of the area of ​​interest (520), the size of the area of ​​interest, information regarding a portion corresponding to the object of interest (602), and information regarding the blur array (692). Method of operation.

5. In Paragraph 1, The operation of acquiring the first partial image (690) above is The method includes an operation to determine whether to perform image processing on the region of interest (601) including the object of interest (602), and The operation of applying the first partial image (810) to the second artificial intelligence neural network model (801) is, Based on the decision to perform image processing on the above-mentioned region of interest (601), the method includes applying the first partial image (810) to the second artificial intelligence neural network model (801). Method of operation.

6. In Paragraph 5, The operation of determining whether to perform image processing on the above-mentioned region of interest (601) is The method includes an operation to determine whether to perform image processing on the region of interest (601) based on at least one of the brightness of the object of interest (602), noise included in the region corresponding to the object of interest (602), or the similarity between a first feature identified from the object of interest (602) and a set second feature. Method of operation.

7. In Paragraph 1, The above method of operation is, The method includes the operation of dynamically allocating memory (130) of the electronic device (101) for processing the first image (510) based on the number of regions of interest (520) identified from the first image (510). Method of operation.

8. In Paragraph 1, The operation of applying the first image (510) to the first artificial intelligence neural network model (701) is, The operation of dividing the first image (510) into a plurality of parts (710) to match the batch size of the first artificial intelligence neural network model (701); and The operation includes sequentially applying parts (710) of the first image (510) to the first artificial intelligence neural network model (701), and The first artificial intelligence neural network model (701) is trained to sequentially perform image processing on parts (710) of the input first image (510). Method of operation.

9. In Paragraph 1, The operation of applying the first partial image (810) to the second artificial intelligence neural network model (801) is, The operation of resizing the first partial image (810) to fit the batch size of the second artificial intelligence neural network model (801); is included. The above second artificial intelligence neural network model (801) is trained to perform image processing such that the first feature identified from the object of interest (602) is similar to the set second feature. Method of operation.

10. A computer-readable, non-transient recording medium having instructions for controlling an electronic device recorded thereon, wherein the instructions are configured to cause the electronic device to perform at least one operation when executed by at least one processor, and the recording medium, A command to acquire a first image (510) using the camera (180) of the electronic device (101); Instructions for performing a first artificial intelligence operation (1110) on the first image (510) using the NPU (410) of the electronic device (101); Instructions for performing a second artificial intelligence operation (1150) using the NPU (410) based on the result of performing the first artificial intelligence operation (1110); and Based on the result of performing the second artificial intelligence operation (1150) above, the method includes a command to obtain a synthetic image for the first image (510), and The instruction to perform the above first artificial intelligence operation (1110) is, A command for identifying a region of interest (520) from the first image (510); and The command includes identifying an object of interest (602) from the area of ​​interest (520) and obtaining a first partial image (690) corresponding to the object of interest (602). The instruction to perform the above second artificial intelligence operation (1150) is, Instructions for applying the first image (510) to a first artificial intelligence neural network model (701) using the NPU (410) of the electronic device (101), and applying the first partial image (810) to a second artificial intelligence neural network model (801) using the NPU (410); and A command for obtaining a second image (730) output from the first artificial intelligence neural network model (701) and a second partial image (820) output from the second artificial intelligence neural network model (801) by performing image processing on the first image (510) using the NPU (410) based on the task sequence input to the NPU (410) from the first artificial intelligence neural network model (701) and by performing image processing on the first partial image (810). The command for acquiring the above composite image is, A command for obtaining a third image by synthesizing the second partial image (901) with the second image (930), comprising Recording media.

11. In an electronic device, Camera (180); At least one processor (120) including an NPU (410); and Includes memory for storing instructions; and The above at least one processor executes the above instructions individually or collectively, thereby causing the electronic device: A first image (510) is obtained using the above camera (180), and Using the above NPU (410), a first artificial intelligence operation (1110) is performed on the first image (510), and Based on the result of performing the first artificial intelligence operation (1110) above, a second artificial intelligence operation (1150) is performed using the NPU (410), and Based on the result of performing the above second artificial intelligence operation (1150), a synthetic image for the above first image (510) is obtained, and The above electronic device is, Identify a region of interest (520) from the first image (510), and Identifying an object of interest (602) from the above region of interest (520), and By obtaining a first partial image (690) corresponding to the object of interest (602), the first artificial intelligence operation is performed, and The above electronic device is, The first image (510) is applied to the first artificial intelligence neural network model (701) using the above NPU (410), and The first partial image (810) is applied to the second artificial intelligence neural network model (801) using the above NPU (410), and Based on the task sequence input to the NPU (410) from the first artificial intelligence neural network model (701) and the second artificial intelligence neural network model (801), image processing is performed on the first image (510) using the NPU (410) to obtain the second image (730) output from the first artificial intelligence neural network model (701) and the second partial image (820) output from the second artificial intelligence neural network model (801) by performing image processing on the first partial image (810). thereby performing the second artificial intelligence operation (1150). The electronic device obtains a third image by synthesizing the second partial image (901) with the second image (930). Electronic device.

12. In Paragraph 11, The above at least one processor executes the above instructions individually or collectively, thereby causing the electronic device: By applying the first image (510) to the third artificial intelligence neural network model (501), the region of interest (520) is identified from the first image, and A blur array (692) for a portion corresponding to the object of interest (602) is obtained from the region of interest (520), and Using the above blur arrays (911, 912), the second partial image (901) is combined with the second image (930). Electronic device.

13. In Paragraph 12, The above at least one processor executes the above instructions individually or collectively, thereby causing the electronic device: By applying the first image (510) to the fourth artificial intelligence neural network model (621), the blur array (692) is obtained based on the probability distribution of the portion corresponding to the object of interest (602) output from the fourth artificial intelligence neural network model (621). Electronic device.

14. In Paragraph 12, The above at least one processor executes the above instructions individually or collectively, thereby causing the electronic device: Obtaining objectification data including at least one of information regarding the coordinates of the area of ​​interest (520), the size of the area of ​​interest, information regarding the part corresponding to the object of interest (602), and information regarding the blur array (692). Electronic device.

15. In Paragraph 11, The above at least one processor executes the above instructions individually or collectively, thereby causing the electronic device: Determining whether to perform image processing on the region of interest (601) including the object of interest (602), and Based on the decision to perform image processing on the above-mentioned region of interest (601), the first partial image (810) is applied to the second artificial intelligence neural network model (801). Electronic device.

Images