Electronic device for performing auto-focusing, non-transitory computer-readable storage medium, and method

Abstract

This electronic device may comprise a camera, at least one processor, and a memory for storing instructions. The instructions may cause, when individually or collectively executed by the at least one processor, the electronic device to: acquire an image through the camera; determine a subject of the image; identify a reference focus position for performing auto-focusing on the determined subject; identify a reference region of the image on the basis of the determined subject; acquire focus positions of portions of the image that divide the reference region of the image; identify a focus position corresponding to the reference focus position among the focus positions; and perform the auto-focusing by adjusting a position of a lens of the camera by using the identified focus position.

Description

Electronic device for performing auto-focusing, non-transient computer-readable storage medium, and method

[0001] The following descriptions relate to an electronic device for performing auto-focusing, a non-transient computer-readable storage medium, and a method.

[0002] The electronic device can perform auto-focusing by adjusting the position of the camera lens according to the focus position of the main subject in the image acquired through the camera.

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0004] An electronic device is provided. The electronic device may include a camera, at least one processor including a processing circuit, and a memory including one or more storage media for storing instructions. The instructions may cause the electronic device to acquire an image through the camera when executed individually or collectively by the at least one processor. The instructions may cause the electronic device to determine a subject of the image when executed individually or collectively by the at least one processor. The instructions may cause the electronic device to identify a reference focus position for performing auto-focusing on the determined subject when executed individually or collectively by the at least one processor. The instructions may cause the electronic device to identify a reference region of the image based on the determined subject when executed individually or collectively by the at least one processor. The above instructions may cause the electronic device to obtain focus positions of parts of the image that divide the reference area of ​​the image when executed individually or collectively by the at least one processor. The above instructions may cause the electronic device to identify a focus position corresponding to the reference focus position among the focus positions when executed individually or collectively by the at least one processor. The above instructions may cause the electronic device to perform auto-focusing by adjusting the position of the camera lens using the identified focus position when executed individually or collectively by the at least one processor.

[0005] A non-transient computer-readable storage medium is provided. The non-transient computer-readable storage medium may store one or more programs. The one or more programs may include instructions that cause the electronic device to acquire an image through the camera when executed by the electronic device having a camera. The one or more programs may include instructions that cause the electronic device to determine the subject of the image when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to identify a reference focus position for performing auto-focusing on the determined subject when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to identify a reference region of the image based on the determined subject when executed by the electronic device. The above one or more programs may include instructions that cause the electronic device to obtain focus positions of portions of the image that divide the reference area of ​​the image when executed by the electronic device. The above one or more programs may include instructions that cause the electronic device to identify a focus position corresponding to the reference focus position among the focus positions when executed by the electronic device. The above one or more programs may include instructions that cause the electronic device to perform auto-focusing by adjusting the position of the camera lens using the identified focus position when executed by the electronic device.

[0006] A method is provided. The method may be performed by an electronic device having a camera. The method may include an operation of acquiring an image through the camera. The method may include an operation of determining a subject of the image. The method may include an operation of identifying a reference focus position for performing auto-focusing on the determined subject. The method may include an operation of identifying a reference area of ​​the image based on the determined subject. The method may include an operation of acquiring focus positions of parts of the image that divide the reference area of ​​the image. The method may include an operation of identifying a focus position among the focus positions that corresponds to the reference focus position. The method may include an operation of performing auto-focusing by adjusting the position of the lens of the camera using the identified focus position.

[0007] Figure 1 is a schematic view of an exemplary electronic device.

[0008] Figure 2 illustrates an example of an environment in which auto-focusing is performed within an electronic device.

[0009] Figure 3 illustrates an example of identifying a reference area based on a main subject determined within an image acquired through a camera.

[0010] FIGS. 4a, FIGS. 4b, and FIGS. 4c illustrate examples of environments for identifying a focus position to perform auto-focusing within an electronic device.

[0011] Figure 5 illustrates an example of adjusting the position of the camera's focus lens.

[0012] Figure 6 is a flowchart illustrating a method for performing auto-focusing within an electronic device.

[0013] FIG. 7 is a block diagram of an electronic device in a network environment according to various embodiments.

[0014] FIG. 8 is a block diagram illustrating a camera module according to various embodiments.

[0015] FIG. 9 is a schematic diagram of an exemplary AI system according to one embodiment.

[0016] Figure 1 is a schematic view of an exemplary electronic device.

[0017] Referring to FIG. 1, the electronic device (101) may include at least one processor (110), a memory (120), and at least one camera (130). The electronic device (101) may include at least a part of the electronic device (701) of FIG. 7 or correspond to at least a part of the electronic device (701) of FIG. 7.

[0018] At least one processor (110) may include a processing circuit. At least one processor (110) may include a single processor or multiple processors. At least one processor (110) may control the memory (120) and / or one or more components (at least one camera (130)) of the electronic device (101). For example, at least one processor (110) may include at least a part of the processor (720) of FIG. 7 or correspond to at least a part of the processor (720) of FIG. 7. For example, at least one processor (110) may include an image signal processor (860) included in the camera module (780) of FIG. 8.

[0019] Memory (120) may store one or more programs configured to be executed individually and / or collectively by at least one processor (110). The one or more programs may include instructions. The instructions may cause an electronic device (101) to perform operations described with reference to FIGS. 2 through 6. Memory (120) may include one or more storage media. At least some of the one or more programs may be available to manage, control, and / or execute focus position calculation logic including a trained model, described below. For example, memory (120) may include at least some of the memory (730) of FIG. 7 or correspond to at least some of the memory (730) of FIG. 7.

[0020] At least one camera (130) can capture (or take) images (e.g., still images) and video. For example, at least one camera (130) may include one or more lenses, image sensors, and / or flashes. For example, the one or more lenses may include a focusing lens. For example, the image sensors may include an image sensor (e.g., a dual-pixel image sensor) for detecting phase differences between optical signals obtained from each pixel of the image. For example, the image sensor may be used for phase detection autofocus (PDAF) to perform autofocusing by detecting the phase differences between the optical signals. For example, at least one camera (130) may include at least a part of the camera module (780) of FIG. 7 or correspond to at least a part of the camera module (780) of FIG. 7.

[0021] The electronic device (101) can perform auto-focusing by adjusting the position of the focus lens of at least one camera (130) according to the focus position of the main subject in the image acquired through at least one camera (130). For example, the focus position may be the position of the focus lens. For example, the electronic device (101) can focus on the main subject by adjusting the position of the focus lens according to the focus position. According to an embodiment, the focus position may be represented in various ways. For example, the focus position may be represented based on the position difference between the current position of the focus lens and the target position of the focus lens. For another example, the focus position may be represented based on the target position of the focus lens. For yet another example, the focus position may be represented based on the depth of focus between the current position of the focus lens and the target position of the focus lens. As another example, the focal position may be represented based on the physical distance (e.g., cm) between the depth corresponding to the current position of the focusing lens and the depth of the main subject corresponding to the target position of the focusing lens. However, it is not limited thereto. For example, the focal position may be represented based on a map corresponding to each part of the image.

[0022] For example, the electronic device (101) can determine the main subject within the image obtained through at least one camera (130) and identify a reference area of ​​the image and a reference focus position for the determined main subject based on the determined main subject. For example, the electronic device (101) can identify the focus positions of the image obtained by detecting phase differences between optical signals obtained from each pixel of the image through the image sensor of at least one camera (130). For example, the electronic device (101) can perform auto-focusing according to the focus position identified using the reference focus position among the focus positions within the reference area of ​​the image. For example, the electronic device (101) can increase the accuracy of auto-focusing by identifying the focus position for performing auto-focusing among the focus positions obtained by detecting the phase differences using the reference area and the reference focus position regarding the main subject of the image. The method of operation of auto-focusing performed within the electronic device (101) is described with reference to FIGS. 2 to 6.

[0023] Figure 2 illustrates an example of an environment in which auto-focusing is performed within an electronic device.

[0024] Referring to FIG. 2, an auto-focusing environment (200) including an image sensor (210), focus position calculation logic (220), a lens driving device (230), and a focus lens (240) is illustrated. For example, the image sensor (210), the lens driving device (230), and the focus lens (240) may be included in at least one camera (130). For example, the focus position calculation logic (220) may be stored in memory (120) and executed by at least one processor (110).

[0025] The image sensor (210) can output information about an image (or video) based on optical signals acquired through the focus lens (240) of at least one camera (130). For example, the information about the image may include information regarding the phase difference of the optical signals. For example, the image sensor (210) may include an image sensor (e.g., a dual-pixel image sensor) for detecting phase differences between optical signals acquired from each pixel of the image. For example, the dual-pixel image sensor may be used for phase detection autofocus (PDAF) to perform autofocusing by detecting the phase differences between the optical signals. For example, the dual-pixel image sensor may identify images having different phases (e.g., a left image and a right image) based on a phase detection pixel that detects light passing in a defined direction through the focus lens (240). According to an embodiment, the information regarding the image may include the images having different phases or a single image in which the images are integrated together. For example, the information regarding the image may be described as information regarding a raw image.

[0026] The focus position calculation logic (220) may include a statistical calculation unit (221), an autofocus calculation unit (222), a focus target position calculation unit (223), a trained model (224), a reference area setting unit (225), and a focus position calculation unit (226). For example, the electronic device (101) may identify a focus position for performing auto-focusing based on obtaining information about the image from the image sensor (210) through the focus position calculation logic (220).

[0027] The electronic device (101) can obtain information about the image from the image sensor (210) through the statistical operation unit (221). The information about the image may include information regarding the phase difference. For example, the electronic device (101) can identify a result value for the statistical operation by performing a statistical operation on each pixel of the image based on the information regarding the phase difference through the statistical operation unit (221). For example, the result value may correspond to the focal position of each pixel of the image. For example, the result value may represent a correlation regarding the phase difference between the optical signals.

[0028] The electronic device (101) can identify focus positions of parts of the image based on the result value through the autofocus calculation unit (222). For example, the focus positions can be identified to perform auto-focusing. For example, the focus positions can be obtained based on the phase differences detected by an image sensor (e.g., a dual-pixel image sensor).

[0029] The electronic device (101) can identify a focus position for performing auto-focusing based on the focus positions through the focus target position calculation unit (223). For example, the electronic device (101) can identify a focus position satisfying a defined condition among the focus positions as the focus position for performing auto-focusing through the focus target position calculation unit (223). For example, the defined condition may be a focus position located at the center of the image among the focus positions. As another example, the defined condition may be a focus position located closest among the focus positions. However, the present disclosure is not limited thereto.

[0030] The electronic device (101) can obtain information about the image from the image sensor (210) through a trained model (224) available for image processing. For example, the trained model (224) may represent one or more calculations to be performed by at least one processor (110). For example, the one or more calculations may be performed to determine a main subject within the image. For example, the main subject may be described as a visual object corresponding to the user's intention for auto-focusing. For example, the trained model (224) may include a computational model designed to simulate the neural activity of an organism and / or a program for performing the calculations of said computational model. For example, the trained model (224) may be an artificial neural network model written in a specified language and including a plurality of layers and / or operations (or calculations). For example, the trained model (224) may include a deep learning model that performs a specific purpose action based on the results of learning training data. For example, the trained model (224) may be described as an on-device AI model running within an electronic device (101). However, the present disclosure is not necessarily limited thereto. For example, the trained model (224) may be an AI model running within an external electronic device (e.g., electronic device (702), electronic device (704), or server (708)). For example, the electronic device (101) may transmit the information regarding the image to the external electronic device. For example, the external electronic device may perform one or more calculations to determine a main subject within the image based on the information regarding the image through the trained model (224).For example, the electronic device (101) can obtain the value of the calculation result from the external electronic device.

[0031] The electronic device (101) can determine the main subject of the image based on the information regarding the image through the trained model (224). For example, the electronic device (101) can identify data obtained from the trained model (224) based on applying the image to the trained model (224). The data may include probability values ​​representing the probability that the main subject exists within each part of the image. For example, the electronic device (101) can determine the main subject of the image based on the probability values ​​included in the data. However, the present disclosure is not limited thereto. For example, the electronic device (101) may determine the main subject within the image based on a preset algorithm.

[0032] The electronic device (101) can identify a reference focus position for performing auto-focusing on the determined subject through a trained model (224). For example, the reference focus position may correspond to the determined main subject.

[0033] The electronic device (101) can identify a reference area of ​​the image based on the determined main subject through the reference area setting unit (225). For example, the reference area of ​​the image may include at least a portion of the image corresponding to the determined main subject. For example, the electronic device (101) can identify the reference area based on the data obtained from the trained model (224) through the reference area setting unit (225). For example, the electronic device (101) can identify the area where the probability value represented by the data exceeds a defined value as the reference area. For example, the image may include a first portion of the image and a second portion of the image. The first portion of the image may correspond to the determined main subject. The second portion of the image may not correspond to the determined main subject. The probability value of the data representing the probability that the main subject exists within the first part of the image may be greater than the probability value of the data representing the probability that the main subject exists within the second part of the image.

[0034] For example, the reference area may be represented in various ways depending on the embodiment. For example, the reference area may be represented as a rectangular area within the image. For example, the reference area may be represented based on point coordinates corresponding to the corners (e.g., top-left corner and bottom-right corner) of the rectangular area. For example, the reference area may be represented based on the reference point coordinates of the rectangle and the width and height of the rectangle. As another example, the reference area may be represented as a pair of one or more reference point coordinates located within the image. As yet another example, the reference area may be represented based on a map corresponding to each part of the image. For example, the map may include values ​​(e.g., points) corresponding to each part of the images.

[0035] The electronic device (101) can obtain the focal positions of parts of the image that divide the reference area of ​​the image through the focal position calculation unit (226). For example, the electronic device (101) can identify a focal position corresponding to the reference focal position among the focal positions through the focal position calculation unit (226). For example, the identified focal position may be the focal position closest to the reference focal position among the focal positions.

[0036] The electronic device (101) can perform auto-focusing by adjusting the position of the focus lens (240) of at least one camera (130) using the identified focus position through the lens driving device (230).

[0037] As described above, the electronic device (101) can increase the accuracy of identifying the main subject within the image by determining the main subject within the image through a trained model (224). For example, the electronic device (101) can increase the accuracy of tracking the movement of the main subject regardless of the field of view of at least one camera (130) by determining the main subject within the image in real time through a trained model (224). For example, the method of determining the main subject through a trained model (224) can increase the accuracy of identifying the main subject within the peripheral area of ​​the image when the main subject is present in the peripheral area of ​​the image, compared to the method of determining the main subject within the central area of ​​the image obtained through at least one camera (130).

[0038] For example, even if the position of the focus lens (240) is skewed toward the foreground and blurring of a subject located in the foreground occurs, or even if the position of the focus lens (240) is skewed toward the foreground and blurring of a subject located in the foreground occurs, the electronic device (101) can determine the main subject within the image through a trained model (224). For example, even if the brightness value of a visual object (e.g., the sun) located in the foreground within the image is identified as a saturated value, the electronic device (101) can increase the accuracy of identifying the main subject within the image by determining the main subject through a trained model (224). For example, even if a visual object located in the foreground within the image (e.g., a plate) is identified, the electronic device (101) can increase the accuracy of identifying the main subject within the image by determining a visual object located in the background (e.g., food on a plate) as the main subject through a trained model (224). For example, the method of determining the main subject through a trained model (224) can increase the accuracy of identifying the main subject regardless of the brightness of the image.

[0039] For example, the electronic device (101) can increase the precision of auto-focusing for the main subject by identifying one focus position for performing auto-focusing among the focus positions of the image identified based on the phase difference between optical signals, based on the main subject determined through the trained model (224).

[0040] Figure 3 illustrates an example of identifying a reference area based on a main subject determined within an image acquired through a camera.

[0041] Referring to FIG. 3, an image (300) obtained through at least one camera (130) is shown. Area (310) is an area corresponding to the central area of ​​the image (300). Area (320) is an area representing the main subject corresponding to the user's intention for auto-focusing. Area (330) is a reference area identified based on the main subject determined by the electronic device (101) through a trained model (224). For example, the electronic device (101) can increase the accuracy of identifying the main subject by determining the area (320) representing the main subject through the trained model (224), even if the main subject does not exist within the area (310) corresponding to the central area of ​​the image (300). For example, the electronic device (101) can increase the precision of auto-focusing on the main subject by identifying an area (330) which is a reference area for selecting focus positions of an image (300) based on the phase difference between optical signals based on a determined area (320).

[0042] FIGS. 4a, FIGS. 4b, and FIGS. 4c illustrate examples of environments for identifying a focus position to perform auto-focusing within an electronic device.

[0043] Referring to FIG. 4a, when an image (e.g., image (300) of FIG. 3) obtained through at least one camera (130) is applied to a trained model (224), an example of data (400a) obtained from the trained model (224) is illustrated. For example, the data (400a) obtained from the trained model (224) can be described as data in a two-dimensional array corresponding to each part of the image (300). For example, the data (400a) may include probability values ​​representing the probability that a main subject exists within each part of the image. For example, the probability values ​​included in the data (400a) are each represented as the brightness of the parts of the image in FIG. 4a. For example, the higher the probability value of the data (400a), the brighter the brightness corresponding to the probability value may be.

[0044] Referring to FIG. 4b, an example of an environment (400b) is shown in which an electronic device (101) determines a main subject of an image (e.g., image (300) of FIG. 3) acquired through at least one camera (130) based on data (400a) shown in FIG. 4a through a trained model (224). The environment (400b) is described as an example of a process in which a reference area setting unit (225) identifies a reference area (e.g., reference area (330) of FIG. 3) for determining the main subject within the image (e.g., image (300) of FIG. 3) based on data (400a) acquired from the trained model (224). The reference area (420) shown in FIG. 4b may correspond to the reference area (e.g., reference area (330) of FIG. 3).

[0045] For example, the electronic device (101) can determine the main subject of the image according to probability values ​​included in the data (400a). For example, the determined main subject may be identified as a part of the image (411), a part of the image (412), a part of the image (413), and a part of the image (414). For example, the electronic device (101) can determine the main subject by identifying the parts corresponding to a defined rank range among the parts of the image (e.g., part of the image (411), part of the image (412), part of the image (413), and part of the image (414)) by identifying the probability values ​​of the data (400a) in order of highest to lowest. For example, the image may include a first part of the image (e.g., part of the image (411)) and a second part of the image (e.g., part of the image (415)). The first part of the image may correspond to the determined main subject. The second part of the image may not correspond to the determined main subject. The probability value of the data representing the probability that the main subject exists within the first part of the image may be greater than the probability value of the data representing the probability that the main subject exists within the second part of the image. In one embodiment, the electronic device (101) may determine the main subject based on the distance between parts corresponding to the defined ranking range.

[0046] For example, the electronic device (101) can identify a reference focus position for performing auto-focusing on the determined main subject.

[0047] For example, the electronic device (101) can identify a reference area (420) of the image based on the part of the image (411), the part of the image (412), the part of the image (413), and the part of the image (414) being determined as the main subject. For example, the reference area (420) of the image may include the part of the image (411), the part of the image (412), the part of the image (413), and the part of the image (414). The electronic device (101) can identify the reference area (420) based on data (400a) obtained from a trained model (224). For example, the electronic device (101) can identify the area where the probability value represented by the data (400a) exceeds a defined value as the reference area (420). In one embodiment, the electronic device (101) may identify a reference area (420) by considering the central area of ​​the image.

[0048] Referring to FIG. 4c, information (400c) regarding focal positions of an image identified according to the detected phase difference is shown, in which an electronic device (101) detects a phase difference between optical signals acquired through a focal lens (240) of at least one camera (130). The area (430) corresponds to the reference area (420) shown in FIG. 4b.

[0049] For example, the electronic device (101) can obtain focus positions of parts of the image that divide a region (430) corresponding to a reference region (420) of the image. In FIG. 4c, each number included in the region (430) may represent the focus positions. For example, the electronic device (101) can identify a focus position corresponding to a reference focus position for a main subject (e.g., part of the image (411), part of the image (412), part of the image (413), and part of the image (414)) determined among the focus positions included in the region (430). For example, the identified focus position may be for performing auto-focusing. For example, the identified focus position may be the focus position closest to the reference focus position among the focus positions. For example, if the reference focus position corresponds to '51', the electronic device (101) can identify '55', which is closest to '51' among each number included in the area (430), as the focus position. For example, the identified focus position may be the focus position of the part (431) of the image.

[0050] Figure 5 illustrates an example of adjusting the position of the camera's focus lens.

[0051] Referring to FIG. 5, an example of an environment (500) in which an electronic device (101) adjusts the position of a focus lens (240) through a lens driving device (230) is illustrated. For example, the electronic device (101) can adjust the position of the focus lens (240) through the lens driving device (230) to a first direction (510) or a second direction (520). For example, the first direction (510) may correspond to a direction toward the image sensor (210). For example, the second direction (520) may correspond to a direction toward the main subject. For example, the electronic device (101) can adjust the focus position to a near view by adjusting the position of the focus lens (240) through the lens driving device (230) to the first direction (510). For example, the electronic device (101) can adjust the focal position to a distant view by adjusting the position of the focal lens (240) to a second direction (520) through the lens driving device (230). For example, the electronic device (101) can perform auto-focusing by adjusting the position of the focal lens (240) of at least one camera (130) using the focal position identified among the focal positions through the lens driving device (230).

[0052] Figure 6 is a flowchart illustrating a method for performing auto-focusing within an electronic device.

[0053] Referring to FIG. 6, in operation 601, at least one processor (110) can acquire an image through at least one camera (130). For example, the image can be acquired through an image sensor (210) of at least one camera (130). For example, the image sensor (210) may include an image sensor (e.g., a dual-pixel image sensor) for detecting phase differences between optical signals acquired from each pixel of the image. For example, the dual-pixel image sensor may be used for phase detection autofocus (PDAF) to perform autofocusing by detecting the phase differences between the optical signals.

[0054] In operation 602, at least one processor (110) can acquire focus locations for each part of the image. For example, the focus locations can be acquired based on the phase differences detected by an image sensor (e.g., a dual-pixel image sensor) within the entire area of ​​the image.

[0055] In operation 603, at least one processor (110) can identify a focus position (Vpd) for performing auto-focusing among the focus positions. For example, at least one processor (110) can identify a focus position among the focus positions that satisfies a defined condition as a focus position (Vpd) for performing auto-focusing. For example, the defined condition may be a focus position located at the center of the image among the focus positions. For another example, the defined condition may be a focus position located closest among the focus positions. However, the present disclosure is not limited thereto.

[0056] In operation 604, at least one processor (110) can determine the main subject of the image. For example, at least one processor (110) can apply the image to a trained model (224) available for image processing based on the acquisition of the image. At least one processor (110) can identify data obtained from the trained model (224) based on the application of the image to the trained model (224). The data may include probability values ​​representing the probability that the main subject exists within each part of the image. At least one processor (110) can determine the main subject of the image according to the probability values ​​included in the data.

[0057] In operation 605, at least one processor (110) can identify a reference focus position (Vref) for performing auto-focusing on the determined main subject. For example, the reference focus position (Vref) may correspond to the determined main subject.

[0058] In operation 606, at least one processor (110) can identify a reference region (Rref) of the image based on the determined main subject. For example, the reference region (Rref) of the image may include at least a portion of the image corresponding to the determined main subject. For example, the image may include a first portion of the image and a second portion of the image. The first portion of the image may correspond to the determined main subject. The second portion of the image may not correspond to the determined main subject. The probability value of the data representing the probability that the main subject exists within the first portion of the image may be greater than the probability value of the data representing the probability that the main subject exists within the second portion of the image.

[0059] In operation 607, at least one processor (110) can identify whether a reference focus position (Vref) is smaller than a threshold value (th). For example, the threshold value (th) can be set differently depending on the embodiment. For example, as the depth of the determined main subject increases as the reference focus position (Vref) increases, at least one processor (110) can perform auto-focusing using the focus position (Vpd) for the precision of focusing on the main subject based on identifying a reference focus position (Vref) that is larger than the threshold value (th). For example, as the depth of the determined main subject decreases as the reference focus position (Vref) decreases, at least one processor (110) can perform auto-focusing using the reference focus position (Vref) for the accuracy of focusing on the main subject based on identifying a reference focus position (Vref) that is smaller than the threshold value (th).

[0060] In operation 608, at least one processor (110) can perform auto-focusing by adjusting the position of the focus lens (240) of at least one camera (130) using a focus position (Vpd) included in the focus positions, based on identifying a reference focus position (Vref) greater than a threshold value (th). For example, at least one processor (110) can terminate the auto-focusing based on identifying the position of the focus lens (240) corresponding to the focus position (Vpd).

[0061] In operation 609, at least one processor (110) can identify whether a reference focus position (Vref) is smaller than a focus position (Vpd). For example, if a reference focus position (Vref) larger than a focus position (Vpd) is identified, the precision of auto-focusing performed using the focus position (Vpd) may be higher than the precision of auto-focusing performed using the reference focus position (Vref). For example, if a reference focus position (Vref) smaller than a focus position (Vpd) is identified, the accuracy of identifying the main subject through auto-focusing performed using the reference focus position (Vref) may be higher than auto-focusing performed using the focus position (Vpd).

[0062] In operation 608, at least one processor (110) can perform auto-focusing by adjusting the position of the focus lens (240) of at least one camera (130) using the focus position (Vpd) based on identifying a reference focus position (Vref) greater than the focus position (Vpd). For example, at least one processor (110) can terminate the performance of auto-focusing based on identifying the position of the focus lens (240) corresponding to the focus position (Vpd).

[0063] In operation 610, at least one processor (110) can obtain the focal positions (Vpd-m) of parts of the image that divide the reference region (Rref) of the image based on identifying a reference focal position (Vref) that is smaller than the threshold value (th) and the focal position (Vpd).

[0064] In operation 611, at least one processor (110) can identify a focus position (Vg) corresponding to a reference focus position (Vref) among the focus positions (Vpd-m). For example, the focus position (Vg) may be the focus position closest to the reference focus position (Vref) among the focus positions (Vpd-m).

[0065] In operation 612, at least one processor (110) can determine whether the focus position (Vg) is included within a range corresponding to the depth of focus. For example, the depth of focus may be described as a reference range set in relation to the focus position for sharp focusing on the main subject. For example, the range may be set to a range corresponding to 'x' times the depth of focus ('x' is a positive number).

[0066] In operation 613, at least one processor (110) can perform auto-focusing by adjusting the position of the focus lens (240) of at least one camera (130) using the focus position (Vg), based on the fact that the focus position (Vg) is identified within the range corresponding to the depth of focus for at least one camera (130). For example, at least one processor (110) can terminate the performance of auto-focusing based on the fact that the position of the focus lens (240) corresponding to the focus position (Vg) is identified.

[0067] In operation 614, at least one processor (110) can mitigate image wobble on the main subject by maintaining the position of the focus lens (240) of at least one camera (130) based on the fact that the focus position (Vg) is identified outside the range.

[0068] In FIG. 6, it has been described that at least one processor (110) performs operations 610, 611, 612, 613, and 614 based on identifying a reference focus position (Vref) smaller than the threshold value (th) and focus position (Vpd) in operations 607 and 609, but this is merely exemplary. For example, operations 607 and 609 may be skipped or omitted. For example, at least one processor (110) may perform operations 612, 613, and 614 by identifying a focus position (Vg) corresponding to the reference focus position (Vref) through operations 610 and 611 without operations 607 and 609.

[0069] In one embodiment, the electronic device (101) can identify whether a condition related to the brightness of the image is satisfied while performing auto-focusing that adjusts the position of the focus lens (240) of at least one camera (130) using the focus position (Vg). For example, the condition related to the brightness may be a condition in which the value representing the brightness is less than a threshold value. For example, the electronic device (101) can perform contrast auto-focusing using the sharpness of the subject identified by adjusting the position of the focus lens (240) based on whether the condition related to the brightness is satisfied. For example, the electronic device (101) can perform contrast auto-focusing by adjusting the position of the focus lens (240) of at least one camera (130) in a direction indicated by a reference focus position (Vref) (e.g., a first direction (510) or a second direction (520) in FIG. 5) based on the condition associated with the brightness being satisfied. For example, since adjusting the position of the focus lens (240) in a random direction causes image wobble during auto-focusing, the electronic device (101) can mitigate the occurrence of image wobble during auto-focusing by adjusting the position of the focus lens (240) in a direction indicated by the reference focus position (Vref). For example, since adjusting the position of the focus lens (240) in a random direction causes a failure of auto-focusing, the electronic device (101) can mitigate the failure of auto-focusing by adjusting the position of the focus lens (240) in a direction indicated by a reference focus position (Vref). However, the present disclosure is not limited thereto. For example, the conditions for performing the contrast auto-focusing can be set in various ways according to the embodiment.For example, the contrast auto-focusing may be performed based on the inability to identify the main subject within the image.

[0070] The electronic device (101) may correspond to the electronic device (701) described with reference to Figure 7 below.

[0071] FIG. 7 is a block diagram of an electronic device in a network environment according to various embodiments.

[0072] Referring to FIG. 7, in a network environment (700), an electronic device (701) may communicate with an electronic device (702) through a first network (798) (e.g., a short-range wireless communication network) or with at least one of an electronic device (704) or a server (708) through a second network (799) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (701) may communicate with the electronic device (704) through a server (708). According to one embodiment, the electronic device (701) may include a processor (720), memory (730), input module (750), sound output module (755), display module (760), audio module (770), sensor module (776), interface (777), connection terminal (778), haptic module (779), camera module (780), power management module (788), battery (789), communication module (790), subscriber identification module (796), or antenna module (797). In some embodiments, at least one of these components (e.g., connection terminal (778)) may be omitted from the electronic device (701), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (776), camera module (780), or antenna module (797)) may be integrated into a single component (e.g., display module (760)).

[0073] The processor (720) can control at least one other component (e.g., hardware or software component) of the electronic device (701) connected to the processor (720) by executing software (e.g., program (740)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (720) can store commands or data received from other components (e.g., sensor module (776) or communication module (790)) in volatile memory (732), process the commands or data stored in volatile memory (732), and store the resulting data in non-volatile memory (734). According to one embodiment, the processor (720) may include a main processor (721) (e.g., central processing unit or application processor) or an auxiliary processor (723) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (701) includes a main processor (721) and an auxiliary processor (723), the auxiliary processor (723) may be configured to use less power than the main processor (721) or to be specialized for a designated function. The auxiliary processor (723) may be implemented separately from the main processor (721) or as part thereof.

[0074] The auxiliary processor (723) may control at least some of the functions or states associated with at least one component of the electronic device (701) (e.g., display module (760), sensor module (776), or communication module (790)) on behalf of the main processor (721) while the main processor (721) is in an inactive (e.g., sleep) state, or together with the main processor (721) while the main processor (721) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (723) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (780) or communication module (790)). According to one embodiment, the auxiliary processor (723) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (701) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (708)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0075] The memory (730) can store various data used by at least one component of the electronic device (701) (e.g., processor (720) or sensor module (776)). The data may include, for example, software (e.g., program (740)) and input or output data for related commands. The memory (730) may include volatile memory (732) or non-volatile memory (734).

[0076] The program (740) may be stored as software in memory (730) and may include, for example, an operating system (742), middleware (744), or an application (746).

[0077] The input module (750) can receive commands or data to be used for a component of the electronic device (701) (e.g., processor (720)) from outside the electronic device (701) (e.g., user). The input module (750) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0078] The sound output module (755) can output an audio signal to the outside of the electronic device (701). The sound output module (755) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0079] The display module (760) can visually provide information to an external (e.g., user) of the electronic device (701). The display module (760) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (760) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0080] The audio module (770) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (770) can acquire sound through the input module (750) or output sound through the sound output module (755) or an external electronic device (e.g., electronic device (702)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (701).

[0081] The sensor module (776) can detect the operating state of the electronic device (701) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (776) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0082] The interface (777) may support one or more specified protocols that can be used for the electronic device (701) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (702)). According to one embodiment, the interface (777) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0083] The connection terminal (778) may include a connector through which the electronic device (701) can be physically connected to an external electronic device (e.g., electronic device (702)). According to one embodiment, the connection terminal (778) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0084] The haptic module (779) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (779) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0085] The camera module (780) can capture still images and video. According to one embodiment, the camera module (780) may include one or more lenses, image sensors, image signal processors, or flashes.

[0086] The power management module (788) can manage power supplied to the electronic device (701). According to one embodiment, the power management module (788) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0087] The battery (789) can supply power to at least one component of the electronic device (701). According to one embodiment, the battery (789) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0088] The communication module (790) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (701) and an external electronic device (e.g., electronic device (702), electronic device (704), or server (708)), and the performance of communication through the established communication channel. The communication module (790) may include one or more communication processors that operate independently of the processor (720) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (790) may include a wireless communication module (792) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (794) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (704) through a first network (798) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (799) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (792) can identify or authenticate the electronic device (701) within a communication network such as the first network (798) or the second network (799) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (796).

[0089] The wireless communication module (792) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (792) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (792) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (792) can support various requirements specified in the electronic device (701), external electronic device (e.g., electronic device (704)), or network system (e.g., second network (799)). According to one embodiment, the wireless communication module (792) can support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0090] An antenna module (797) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (797) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (797) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (798) or a second network (799), may be selected from the plurality of antennas, for example, by a communication module (790). A signal or power may be transmitted or received between the communication module (790) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (797).

[0091] According to various embodiments, the antenna module (797) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0092] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0093] According to one embodiment, commands or data may be transmitted or received between the electronic device (701) and an external electronic device (704) through a server (708) connected to a second network (799). Each of the external electronic devices (702, or 704) may be the same or a different type of device as the electronic device (701). According to one embodiment, all or part of the operations performed on the electronic device (701) may be performed on one or more of the external electronic devices (702, 704, or 708). For example, if the electronic device (701) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (701) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (701). The electronic device (701) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (701) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (704) may include an Internet of Things (IoT) device. The server (708) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (704) or the server (708) may be included within the second network (799).The electronic device (701) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0094] FIG. 8 is a block diagram illustrating a camera module according to various embodiments.

[0095] Referring to the block diagram (800) illustrated in FIG. 8, the camera module (780) may include a lens assembly (810), a flash (820), an image sensor (830), an image stabilizer (840), a memory (850) (e.g., a buffer memory), or an image signal processor (860).

[0096] A lens assembly (810) can collect light emitted from a subject that is the subject of image capture. The lens assembly (810) may include one or more lenses. According to one embodiment, a camera module (780) may include a plurality of lens assemblies (810). In this case, the camera module (780) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (810) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties different from the lens properties of other lens assemblies. The lens assembly (810) may include, for example, a wide-angle lens or a telephoto lens.

[0097] The flash (820) may emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (820) may include one or more light-emitting diodes (e.g., RGB (red-green-blue) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp.

[0098] The image sensor (830) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (810) into an electrical signal. According to one embodiment, the image sensor (830) may include, for example, one image sensor selected from image sensors with different properties such as an RGB sensor, a BW (black and white) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same properties, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (830) may be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0099] The image stabilizer (840) may move at least one lens or image sensor (830) included in the lens assembly (810) in a specific direction or control the operational characteristics of the image sensor (830) (e.g., adjusting read-out timing, etc.) in response to the movement of the camera module (780) or the electronic device (701) containing it. This allows for compensating for at least some of the negative effects caused by the movement on the image being captured. According to one embodiment, the image stabilizer (840) may detect such movement of the camera module (780) or the electronic device (701) using a gyroscope sensor (not shown) or an accelerometer sensor (not shown) placed inside or outside the camera module (780). According to one embodiment, the image stabilizer (840) may be implemented, for example, as an optical image stabilizer.

[0100] The memory (850) may temporarily store at least a portion of the image acquired through the image sensor (830) for the next image processing operation. For example, if image acquisition by the shutter is delayed or multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (850), and the corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (760). Subsequently, when a specified condition is satisfied (e.g., user input or system command), at least a portion of the original image stored in the memory (850) may be acquired and processed, for example, by an image signal processor (860). According to one embodiment, the memory (850) may be configured as at least a portion of the memory (730) or as a separate memory that operates independently thereof.

[0101] The image signal processor (860) can perform one or more image processing operations on an image acquired through the image sensor (830) or an image stored in memory (850). One or more of the above image processing may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softing). Additionally or generally, the image signal processor (860) may perform control (e.g., exposure time control, or readout timing control, etc.) over at least one of the components included in the camera module (780) (e.g., image sensor (830)). The image processed by the image signal processor (860) may be stored back in memory (850) for further processing or provided to an external component of the camera module (780) (e.g., memory (730), display module (760), electronic device (702), electronic device (704), or server (708)). According to one embodiment, the image signal processor (860) is at least part of the processor (720). It may be configured as a separate processor that operates independently of the processor (720). If the image signal processor (860) is configured as a separate processor from the processor (720), at least one image processed by the image signal processor (860) may be displayed through the display module (760) as is or after additional image processing by the processor (720).

[0102] According to one embodiment, the electronic device (701) may include a plurality of camera modules (780), each having different attributes or functions. In this case, for example, at least one of the plurality of camera modules (780) may be a wide-angle camera and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (780) may be a front camera and at least another may be a rear camera.

[0103] FIG. 9 is a schematic diagram of an exemplary AI system according to one embodiment.

[0104] Referring to FIG. 9, the AI ​​system (900) may include an input / output interface (910), an AI (artificial intelligence) framework (920), a generative AI model (930), an application / service component (980), and / or a knowledge repository (990).

[0105] The input / output interface (910) can receive input. The input may include user input and / or data obtained or generated by an electronic device (e.g., the electronic device (101) or electronic device (701) described above). The data may include images, videos, and / or sensor data generated by at least one processor of the electronic device (e.g., at least one processor (110) or processor (720)), such as illuminance data around the electronic device obtained from a sensor or sensor hub (e.g., auxiliary processor (723), attitude data (or orientation data) of the electronic device, temperature inside the electronic device (e.g., display) or temperature of at least one processor (110), size information of the display area of ​​the display, and / or images obtained through an image sensor of the electronic device (e.g., included in a camera module (780)). The user input may include natural language, touch data obtained through a touch circuit included within the display panel (e.g., used to identify input from a finger and / or stylus), an image displayed (and / or to be displayed) on the display panel, and / or video. By example, without limitation, the user input may be received by an input / output interface (910) along with context information. The context information may be described as additional information obtained in relation to the user input. The context information may be related to the state at the time the user input is received (e.g., the state of the electronic device and / or the state of the surroundings of the electronic device (e.g., user state)). For example, the context information may include information about one or more software applications executed within the electronic device at the time the user input is received.For example, the above situation information may include information about the location of the electronic device (or the location of the user of the electronic device) at the time the user input is received. For example, the user input may be integrated with the situation information. For example, the user input with the situation information integrated as the input may be received by the input / output interface (910).

[0106] The input / output interface (910) may transmit (or provide) an output. The output may include a result (or result information) generated or obtained by the AI ​​system (900) based on at least part of the input. The format of the output may vary. For example, the output may include natural language. For example, the output may include content (e.g., media content and / or multimedia content). For example, the output may include actions related to the user of the electronic device. For example, the output may have a format according to the user settings of the electronic device.

[0107] The input / output interface (910) can be described as a user question / response interface (910).

[0108] The AI ​​framework (920) can be used to obtain information (or data) about the input from the input / output interface (910) and to control one or more components related to the AI ​​system (900) using the obtained information.

[0109] For example, a prompt design component (921) within an AI framework (920) can generate or obtain prompts for a generative AI model (930) (e.g., including a large language model (LLM) or a large multimodal model (LMM)) using the acquired information. For example, the prompt design component (921) may be described as an AI component that uses a learning algorithm and / or a neural network to provide prompts that are enhanced over time. For example, the prompt design component (921) can generate or obtain prompts by accessing a knowledge component (e.g., a knowledge repository (990)) containing user preference data, a prompt library, and / or prompt examples using the acquired information. The generated prompts may be provided to the generative AI model (930) (e.g., including an LLM or LMM).

[0110] For example, an API / plugin management component (922) within the AI ​​framework (920) may be used to support communication for additional information requested (or induced) in relation to the prompt provided (or to be provided) to the generative AI model (930). For example, the API / plugin management component (922) may be used to create or establish channels for communication with various data sources (e.g., knowledge repository (990)). For example, the API / plugin management component (922) may support access to at least some of the data sources. For example, the API / plugin management component (922) may be used to request other components (e.g., application / service component (980)) that perform feedback (or response) according to the prompt. As an example without limitation, information obtained (or generated) through the API / plugin management component (922) may be provided to the prompt design component (921) for the creation of the prompt. As an example that is not limited, information obtained (or generated) through the API / plugin management component (922) can be provided to the generative AI model (930).

[0111] For example, an improvement component (923) within the AI ​​framework (920) can at least partially tune (or adjust) (or change) the result (e.g., content) obtained (or output) from the generative AI model (930). For example, the improvement component (923) can determine or verify whether the content obtained from the generative AI model (930) is related to the input. For example, the improvement component (923) can determine or verify whether the content obtained from the generative AI model (930) contains biased content. For example, the improvement component (923) can determine or verify whether the content obtained from the generative AI model (930) contains harmful content. For example, the improvement component (923) can support or assist in performing additional processing to improve the content obtained from the generative AI model (930). For example, the improvement component (923) may support providing a hint to the user to improve the content.

[0112] A generative AI model (930) can be described as an artificial intelligence neural network that generates feedback in response to a prompt. For example, the feedback may include additional data and / or information relative to the prompt, but relative to the prompt. For example, the feedback may include new content relative to the prompt. For example, the generative AI model (930) may include a model that generates images and / or a model that generates language. For example, the model that generates images may include a generative adversarial network (GAN) and / or a variational autoencoder (VAE). For example, the model that generates images may include a diffusion-based generative model (e.g., a transformer VAE). For example, the model that generates language may include CHAT-GPT 3 and / or CHAT-GPT 4. For example, the generative AI model (930) may include an LMM that generates the feedback by recognizing text, images, and / or speech.

[0113] As an example without limitation, the AI ​​framework (920) and / or generative AI model (930) may be included within an AI module (e.g., including a processing circuit) within the electronic device. For example, the AI ​​module may be operatively coupled with at least one processor of the electronic device (e.g., at least one processor (110) or processor (720)). For example, the AI ​​module may be operatively coupled with a display driving circuit of the electronic device. For example, the AI ​​module may be operatively coupled with a sensor hub of the electronic device for one or more sensors within the electronic device.

[0114] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains.

[0115] As described above, an electronic device (e.g., electronic device (101)) may include a camera (e.g., at least one camera (130)), at least one processor (e.g., at least one processor (110)) including a processing circuit; and a memory (e.g., memory (120)) that stores instructions and includes one or more storage media. When the instructions are executed individually or collectively by the at least one processor, the instructions include: acquiring an image through the camera; determining a subject of the image; identifying a reference focus position for performing auto-focusing on the determined subject; identifying a reference region of the image based on the determined subject; acquiring focus positions of parts of the image that divide the reference region of the image; and identifying a focus position among the focus positions that corresponds to the reference focus position. And the electronic device may be made to perform auto-focusing by adjusting the position of the lens of the camera (e.g., focus lens (240)) using the identified focus position.

[0116] For example, the identified focus position may be the focus position closest to the reference focus position among the focus positions.

[0117] For example, the above instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to: apply the image to a trained model available for image processing based on the acquisition of the image; identify data acquired from the trained model based on the application of the image to the trained model, wherein the data includes probability values ​​representing the probability that the subject exists within each part of the image; and determine the subject of the image according to the probability values ​​included in the data.

[0118] For example, the above image includes a first part of the image and a second part of the image, wherein the first part of the image corresponds to the determined subject and the second part of the image does not correspond to the determined subject, and the probability value of the data representing the probability that the subject exists within the first part of the image may be greater than the probability value of the data representing the probability that the subject exists within the second part of the image.

[0119] For example, the reference area of ​​the above image may include at least a portion of the image corresponding to the determined subject.

[0120] For example, the camera includes an image sensor (e.g., image sensor (210)) for detecting phase differences between signals obtained from each pixel of the image, and the focus positions can be obtained based on the detected phase differences.

[0121] For example, the identified focus position may be a first focus position included among the focus positions. When the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to perform the auto-focusing by adjusting the position of the lens of the camera using the first focus position based on the identification of the reference focus position smaller than a threshold value; and to perform the auto-focusing by adjusting the position of the lens of the camera using a second focus position included among the focus positions based on the identification of the reference focus position larger than the threshold value.

[0122] For example, the identified focus position may be a first focus position included among the focus positions. When the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to perform the auto-focusing by adjusting the position of the lens of the camera using the first focus position based on identifying the reference focus position smaller than the second focus position included among the focus positions; and to perform the auto-focusing by adjusting the position of the lens of the camera using the second focus position based on identifying the reference focus position larger than the second focus position.

[0123] For example, when the above instructions are executed individually or collectively by the at least one processor: the electronic device may perform the auto-focusing by adjusting the position of the lens of the camera using the identified focus position based on the identified focus position being identified within a range corresponding to the depth of focus for the camera; and maintain the position of the lens of the camera based on the identified focus position being identified outside the range.

[0124] For example, when the above instructions are executed individually or collectively by the at least one processor: identifying that a condition associated with the brightness of the image is satisfied while performing the auto-focusing that adjusts the position of the lens of the camera using the identified focus position; and based on the satisfaction of the condition, causing the electronic device to perform the auto-focusing by adjusting the position of the lens of the camera in a direction indicated by the reference focus position.

[0125] A non-transient computer-readable storage medium as described above may store one or more programs. The one or more programs may include instructions that, when executed by an electronic device (e.g., electronic device (101)) having a camera (e.g., at least one camera (130)), acquire an image through the camera; determine a subject of the image; identify a reference focus position for performing auto-focusing on the determined subject; identify a reference area of ​​the image based on the determined subject; acquire focus positions of parts of the image that divide the reference area of ​​the image; identify a focus position among the focus positions that corresponds to the reference focus position; and cause the electronic device to perform auto-focusing by adjusting the position of the lens of the camera (e.g., focus lens (240)) using the identified focus position.

[0126] For example, the identified focus position may be the focus position closest to the reference focus position among the focus positions.

[0127] For example, the above one or more programs may include instructions that, when executed by the electronic device: apply the image to a trained model available for image processing based on the acquisition of the image; identify data acquired from the trained model based on the application of the image to the trained model, wherein the data includes probability values ​​representing the probability that the subject exists within each part of the image; and cause the electronic device to determine the subject of the image according to the probability values ​​included in the data.

[0128] For example, the above image includes a first part of the image and a second part of the image, wherein the first part of the image corresponds to the determined subject and the second part of the image does not correspond to the determined subject, and the probability value of the data representing the probability that the subject exists within the first part of the image may be greater than the probability value of the data representing the probability that the subject exists within the second part of the image.

[0129] For example, the reference area of ​​the above image may include at least a portion of the image corresponding to the determined subject.

[0130] For example, the camera includes an image sensor (e.g., image sensor (210)) for detecting phase differences between signals obtained from each pixel of the image, and the focus positions can be obtained based on the detected phase differences.

[0131] For example, the identified focus position may be a first focus position included among the focus positions. The one or more programs may include instructions that cause the electronic device to perform the auto-focusing by adjusting the position of the lens of the camera using the first focus position based on the identification of the reference focus position smaller than a threshold value; and to perform the auto-focusing by adjusting the position of the lens of the camera using a second focus position included among the focus positions based on the identification of the reference focus position larger than the threshold value.

[0132] For example, the identified focus position may be a first focus position included among the focus positions. The one or more programs may include instructions that cause the electronic device to perform the auto-focusing by adjusting the position of the lens of the camera using the first focus position based on identifying the reference focus position smaller than the second focus position included among the focus positions; and to perform the auto-focusing by adjusting the position of the lens of the camera using the second focus position based on identifying the reference focus position larger than the second focus position.

[0133] For example, the above one or more programs may include instructions that, when executed by the electronic device: perform the auto-focusing by adjusting the position of the lens of the camera using the identified focus position based on the identified focus position being identified within a range corresponding to the depth of focus for the camera; and cause the electronic device to maintain the position of the lens of the camera based on the identified focus position being identified outside the range.

[0134] The method described above may be performed by an electronic device (e.g., electronic device (101)) having a camera (e.g., at least one camera (130)). The method may include the operation of acquiring an image through the camera; the operation of determining a subject of the image; the operation of identifying a reference focus position for performing auto-focusing on the determined subject; the operation of identifying a reference area of ​​the image based on the determined subject; the operation of acquiring focus positions of parts of the image that divide the reference area of ​​the image; the operation of identifying a focus position among the focus positions that corresponds to the reference focus position; and the operation of performing auto-focusing by adjusting the position of the lens of the camera using the identified focus position.

[0135] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs.

[0136] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0137] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0138] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0139] Various embodiments of the present document may be implemented as software (e.g., program (740)) comprising one or more instructions stored in a storage medium (e.g., internal memory (736) or external memory (738)) readable by a machine (e.g., electronic device (701)). For example, a processor (e.g., processor (720)) of the machine (e.g., electronic device (701)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0140] According to one embodiment, the method according to the various embodiments disclosed herein may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0141] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device, camera; At least one processor including a processing circuit; and Memory that stores instructions and includes one or more storage media, When the above instructions are executed individually or collectively by the at least one processor: Acquire an image through the above camera; Determine the subject of the above image; Identifying a reference focus position for performing auto-focusing on the above-determined subject; Identifying a reference area of ​​the image based on the determined subject above; Obtaining focal positions of parts of the image that divide the reference area of ​​the image; Identifying a focus position among the above focus positions that corresponds to the reference focus position; and To perform the auto-focusing by adjusting the position of the camera lens using the identified focus position, The above electronic device, causing, Electronic device.

2. In Claim 1, The identified focus position is the focus position closest to the reference focus position among the focus positions, Electronic device.

3. In Claim 1, When the above instructions are executed individually or collectively by the at least one processor: Based on the acquisition of the above image, the image is applied to a trained model available for image processing; Based on applying the image to the above-mentioned trained model, data obtained from the above-mentioned trained model is identified, and the data includes probability values ​​representing the probability that the subject exists within each part of the above-mentioned image; and To determine the subject of the image according to the probability values ​​included in the data above, The above electronic device, causing, Electronic device.

4. In Claim 3, The above image includes a first part of the above image and a second part of the above image, and The first part of the above image corresponds to the determined subject, and The second portion of the above image does not correspond to the determined subject, and The probability value of the data representing the probability that the subject exists within the first part of the image is greater than the probability value of the data representing the probability that the subject exists within the second part of the image. Electronic device.

5. In Claim 1, The reference region of the above image comprises at least a portion of the image corresponding to the determined subject. Electronic device.

6. In Claim 1, The camera includes an image sensor for detecting phase differences between signals obtained from each pixel of the image, and The above focus positions are obtained based on the detected phase differences, Electronic device.

7. In Claim 1, The identified focus position is a first focus position included in the focus positions, and When the above instructions are executed individually or collectively by the at least one processor: Based on identifying the reference focus position smaller than a threshold value, the auto-focusing is performed by adjusting the position of the lens of the camera using the first focus position; and Based on identifying a reference focus position greater than the threshold value, the auto-focusing is performed by adjusting the position of the lens of the camera using a second focus position included in the focus positions. The above electronic device, causing, Electronic device.

8. In Claim 1, The identified focus position is a first focus position included in the focus positions, and When the above instructions are executed individually or collectively by the at least one processor: Based on identifying a reference focus position smaller than a second focus position included in the above focus positions, the auto-focusing is performed by adjusting the position of the lens of the camera using the first focus position; and Based on the identification of a reference focus position that is larger than the second focus position, the auto-focusing is performed by adjusting the position of the lens of the camera using the second focus position. The above electronic device, causing, Electronic device.

9. In Claim 1, When the above instructions are executed individually or collectively by the at least one processor: Based on the fact that the identified focus position is identified within a range corresponding to the depth of focus for the camera, the auto-focusing is performed by adjusting the position of the lens of the camera using the identified focus position; and Based on the fact that the identified focus position is identified outside the range, to maintain the position of the lens of the camera, The above electronic device, causing, Electronic device.

10. In Claim 1, When the above instructions are executed individually or collectively by the at least one processor: Identifying that a condition associated with the brightness of the image is satisfied while performing the auto-focusing that adjusts the position of the lens of the camera using the identified focus position; and Based on the satisfaction of the above conditions, the auto-focusing is performed by adjusting the position of the lens of the camera in the direction indicated by the reference focus position. The above electronic device, causing, Electronic device.

11. In a non-transient computer-readable storage medium storing one or more programs, said one or more programs, when executed by an electronic device having a camera: Acquire an image through the above camera; Determine the subject of the above image; Identifying a reference focus position for performing auto-focusing on the above-determined subject; Identifying a reference area of ​​the image based on the determined subject above; Obtaining focal positions of parts of the image that divide the reference area of ​​the image; Identifying a focus position among the above focus positions that corresponds to the reference focus position; and To perform the auto-focusing by adjusting the position of the camera lens using the identified focus position, Instructions including those that cause the above electronic device Non-transient computer-readable storage media.

12. In Claim 11, The identified focus position is the focus position closest to the reference focus position among the focus positions, Non-transient computer-readable storage media.

13. In Claim 11, When one or more of the above programs are executed by the electronic device: Based on the acquisition of the above image, the image is applied to a trained model available for image processing; Based on applying the image to the above-mentioned trained model, data obtained from the above-mentioned trained model is identified, and the data includes probability values ​​representing the probability that the subject exists within each part of the above-mentioned image; and To determine the subject of the image according to the probability values ​​included in the data above, Instructions including those that cause the above electronic device Non-transient computer-readable storage media.

14. In Claim 13, The above image includes a first part of the above image and a second part of the above image, and The first part of the above image corresponds to the determined subject, and The second portion of the above image does not correspond to the determined subject, and The probability value of the data representing the probability that the subject exists within the first part of the image is greater than the probability value of the data representing the probability that the subject exists within the second part of the image. Non-transient computer-readable storage media.

15. A method performed by an electronic device having a camera, The operation of acquiring an image through the above camera; Action of determining the subject of the above image; An operation to identify a reference focus position for performing auto-focusing on the above-determined subject; An operation to identify a reference area of ​​the image based on the determined subject; An operation to obtain focus positions of parts of the image that divide the reference area of ​​the image; An operation of identifying a focus position corresponding to the reference focus position among the above focus positions; and The operation of performing the auto-focusing by adjusting the position of the lens of the camera using the identified focus position, method.

Images