Electronic device, method, and non-transitory computer-readable storage medium for generating panoramic image

Abstract

This electronic device may comprise at least one sensor, at least one camera, a memory storing instructions, and at least one processor. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to: drive the at least one camera to generate a panoramic image; identify motion data of the electronic device; obtain a first image and a second image while generating the panoramic image; compare the second image with the first image; identify a portion of the first image to be stitched and a portion of the second image to be stitched; obtain information for stitching the first image and the second image by using the motion data, the first image, and the second image on the basis of a failure of the identification; and obtain stitched portions of the first image and the second image by providing the information to a trained model.

Description

Electronic device, method, and non-transient computer-readable storage medium for generating panoramic images

[0001] The following descriptions relate to an electronic device, a method, and a non-transient computer-readable storage medium for generating a panoramic image.

[0002] The electronic device may include at least one camera. For example, the electronic device may acquire images by driving the at least one camera. For example, the electronic device may stitch at least a portion of the images while acquiring the images. For example, the electronic device may generate an image in which at least a portion of the images are stitched. For example, the stitched image may be referred to as a panoramic image.

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0004] The electronic device may include at least one sensor. The electronic device may include at least one camera. The electronic device may include a memory that stores instructions and includes one or more storage media. The electronic device may include at least one processor that includes a processing circuit. The instructions may cause the electronic device to drive the at least one camera to generate a panoramic image when the at least one processor is executed individually or collectively. The instructions may cause the electronic device to identify motion data of the electronic device through the at least one sensor when the at least one processor is executed individually or collectively. The instructions may cause the electronic device to acquire a first image and a second image following the first image while generating the panoramic image when the at least one processor is executed individually or collectively. The above instructions may cause the electronic device to compare the second image with the first image while generating the panoramic image, when the at least one processor is executed individually or collectively. The above instructions may cause the electronic device to perform identification of the portion of the first image to be stitched to generate the panoramic image and the portion of the second image to be stitched to generate the panoramic image, based on the result of the comparison, while generating the panoramic image, when the at least one processor is executed individually or collectively.The above instructions may cause the electronic device, when the at least one processor is executed individually or collectively, to acquire information for stitching the first image and the second image using the motion data identified during the acquisition of the first image and the second image based on the failure of identification while generating the panoramic image, using the first image and the second image. The above instructions may cause the electronic device, when the at least one processor is executed individually or collectively, to acquire the portion where the first image and the second image are stitched to be included in the panoramic image by providing the information to a trained model while generating the panoramic image.

[0005] The electronic device may include at least one sensor. The electronic device may include at least one camera. The electronic device may include a display. The electronic device may include a memory that stores instructions and includes one or more storage media. The electronic device may include at least one processor that includes a processing circuit. The instructions may cause the electronic device to display a first image among a plurality of images through the display while acquiring a plurality of images through the at least one camera based on a panoramic shooting mode when the at least one processor is executed individually or collectively. The instructions may cause the electronic device to identify whether it can determine the relative position of the second image with respect to the first image based on the first image and the second image following the first image among the plurality of images while the electronic device displays the first image through the display when the at least one processor is executed individually or collectively. The above instructions may cause the electronic device to determine the relative position of the second image with respect to the first image based on motion data of the electronic device acquired through the at least one sensor, depending on the failure to determine the relative position of the second image with respect to the first image when the at least one processor is executed individually or collectively. The above instructions may cause the electronic device to display, through the display, a first stitched image in which the second image is at least partially superimposed on the first image, using the relative position of the second image.The above instructions may cause the electronic device, when the at least one processor is executed individually or collectively, to identify whether it can determine the relative position of the third image with respect to the second image based on the second image and the third image following the second image among the plurality of images while the first stitched image is displayed through the display. The above instructions may cause the electronic device, when the at least one processor is executed individually or collectively, to provide at least a portion of the first stitched image to a trained model stored in memory, depending on the success of determining the relative position of the third image with respect to the second image. The above instructions may cause the electronic device, when the at least one processor is executed individually or collectively, to use the trained model to generate an image representing an area where the second image is at least partially superimposed on the first image. The above instructions may cause the electronic device to display, through the display, a second stitched image in which the third image is at least partially superimposed on the first stitched image in which at least a portion is replaced by the generated image, when the at least one processor is executed individually or collectively.

[0006] FIGS. 1a to 1c illustrate an example of a method for generating a panoramic image in a panoramic shot mode.

[0007] FIG. 2 illustrates an example of a method for generating a corrected panoramic image based on an image generated using a trained model.

[0008] Figure 3 is an exemplary schematic view of an electronic device.

[0009] Figure 4 illustrates an example of the operation flow for generating a panoramic image.

[0010] FIG. 5 illustrates an example of a flow of operations for determining a generation area while performing the arrangement of images to generate a panoramic image.

[0011] FIG. 6a illustrates examples of cases where the relative position of an image cannot be determined.

[0012] FIG. 6b illustrates an example of a method for identifying regions to be generated using a trained model while performing the arrangement of images.

[0013] FIG. 7 illustrates an example of a flow of operations for determining a generation area while performing stitching of images to generate a panoramic image.

[0014] FIG. 8a illustrates an example of a method for identifying the boundary between parts of images to be stitched together.

[0015] FIG. 8b illustrates an example of a method for identifying regions to be generated using a trained model while performing stitching of images.

[0016] FIG. 9 illustrates an example of a flow of operations for determining a generation area in a stitched image to generate a panoramic image.

[0017] FIG. 10a illustrates an example of a method for identifying a reference field of view region for a stitched image.

[0018] FIG. 10b illustrates an example of a method for identifying regions to be generated using a trained model in a stitched image.

[0019] FIG. 11 illustrates an example of the operation flow for a method of generating an image corresponding to a generated area using a trained model.

[0020] FIG. 12a illustrates an example of a method for generating an image corresponding to a generated region using a trained model.

[0021] FIG. 12b illustrates an example of a method for generating information for stitching images to be used as input to a trained model.

[0022] FIG. 12c illustrates an example of a method for generating an image corresponding to a generated region using a trained model by providing information for stitching images.

[0023] FIG. 13 illustrates an example of a method for obtaining stitched parts of images using a trained model based on the failure to identify parts of the images to be stitched in order to generate a panoramic image.

[0024] FIG. 14 is a block diagram of an electronic device in a network environment according to various embodiments.

[0025] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit the scope of other embodiments. A singular expression may include a plural expression unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by those skilled in the art described in this disclosure. Terms used in this disclosure that are defined in a general dictionary may be interpreted as having the same or similar meaning as they have in the context of the relevant technology, and are not to be interpreted in an ideal or overly formal sense unless explicitly defined in this disclosure. In some cases, even terms defined in this disclosure are not to be interpreted to exclude the embodiments of this disclosure.

[0026] In the various embodiments of the present disclosure described below, a hardware-based approach is described as an example. However, since the various embodiments of the present disclosure include techniques using both hardware and software, the various embodiments of the present disclosure do not exclude a software-based approach.

[0027] Additionally, in this disclosure, expressions of "greater than" or "less than" may be used to determine whether a specific condition is satisfied or fulfilled; however, this is merely for the purpose of expressing an example and does not exclude descriptions of "greater than" or "less than." Conditions described as "greater than" may be replaced with "greater than," conditions described as "less than" may be replaced with "less than," and conditions described as "greater than and less than" may be replaced with "greater than and less than." Furthermore, "A" to "B" below refer to at least one of the elements from A (including A) to B (including B).

[0028] FIGS. 1a to 1c illustrate an example of a method for generating a panoramic image in a panoramic shot mode.

[0029] FIGS. 1A and 1B illustrate examples (110, 120, 130, 140, 150, 160) of a screen displayed through the display of the electronic device (101) while the electronic device (101) acquires images to generate a panoramic image in a panoramic shooting mode. FIG. 1C illustrates an example of a panoramic image (170) generated using images acquired by the electronic device (101) in a panoramic shooting mode.

[0030] For example, the electronic device (101) may acquire images by driving at least one camera of the electronic device (101) within the panoramic shooting mode of the executed camera application, and may generate a panoramic image using the acquired images. In the above example, the panoramic shooting mode is merely illustrative for convenience of explanation and the present disclosure is not limited thereto. For example, the panoramic shooting mode may be an example of a function for the electronic device (101) to stitch (or blend, synthesize, or combine) images acquired in real time. For example, acquiring a panoramic image within the panoramic shooting mode may be referred to as real-time image stitching, a real-time image stitching function, or a real-time image stitching technique. Additionally, the panoramic shooting mode is exemplified as a mode (or function) of the camera application, but the present disclosure is not limited thereto. For example, the above-mentioned panoramic shooting mode may be an example of a mode (or function) of another application (e.g., an image processing application, a video call application, a remote shooting application). For convenience of explanation, it is assumed below that the electronic device (101) generates a panoramic image within the above-mentioned panoramic shooting mode of the executed camera application.

[0031] Referring to FIGS. 1a and 1b, the electronic device (101) may display a screen (103) through a display while acquiring an image to generate a panoramic image. For example, the screen (103) may include a first screen portion (105) in which a preview image representing a portion of the external environment included in the acquired image is displayed, a second screen portion (107) in which a thumbnail image representing the panoramic image being generated is displayed, and a button (109) for controlling the capture of the image. The screen (103) of FIGS. 1a and 1b is merely an exemplary example for convenience of explanation and is not limited thereto. For example, the screen (103) may further include a button (or icon, executable object) for changing a setting for capturing the image.

[0032] For example, the first screen portion (105) may be used to display a preview image representing a portion of an external environment that is included (or represented) by an image obtained through at least one camera of the electronic device (101). For example, the first screen portion (105) may be referred to as a preview screen, a preview portion, or an external environment display portion.

[0033] For example, the second screen portion (107) may be used to display a thumbnail image representing a panoramic image being generated using images acquired within the panoramic shooting mode. For example, the thumbnail image may be an image in which the panoramic image being generated is displayed in a reduced size. As a non-limiting example, the second screen portion (107) may be located in a part of the first screen portion (105). For example, the part of the first screen portion (105) where the second screen portion (107) is located may include the center area of ​​the first screen portion (105). For example, the second screen portion (107) may represent an entire angle of view area defined based on the first image acquired (hereinafter, the first image) and a preset angle of view within the panoramic shooting mode. As a non-limiting example, the preset angle of view may represent an angle of view (e.g., about 180°) defined from the location where the first image was taken to generate the panoramic image. For example, the entire field of view area may include a reference field of view area. For example, the reference field of view area may be defined by a first reference field of view in a first direction (e.g., vertical or vertical direction) defined by the initial image and a second reference field of view in a second direction (e.g., horizontal or horizontal direction) perpendicular to the first direction defined by the panoramic image being generated. Specific details regarding this may be referenced in FIG. 10a below. For example, the second screen portion (107) may include an indicator to notify the user of a change in the posture (or movement, direction, position) of the electronic device (101) for generating a panoramic image.

[0034] For example, the button (109) may be a button (or icon, executable object) of a camera application for starting, stopping, resuming, or stopping the capture of an image. For example, the electronic device (101) may start (or initiate) the capture of an image, stop the capture of a started image, resume the capture of a stopped image, or stop the capture of an image by identifying an input to the button (109).

[0035] Referring to example (110), the electronic device (101) can start capturing an image to create a panoramic image by identifying an input to a button (109). For example, the electronic device (101) can acquire a first image through at least one camera when the capture of the image begins. For example, the first image may be referred to as the initial image. For example, the electronic device (101) may display a first preview image (111) representing the first image through a first screen portion (105) of the screen (103). For example, the electronic device (101) may display a first thumbnail image (113) representing a panoramic image being created using the first image through a second screen portion (107) of the screen (103). For example, the first thumbnail image (113) may represent a first panoramic image substantially identical to the first image. For example, the electronic device (101) may display a box (115) indicating the location (or angle of view, area) of the first image in a reference angle of view area defined based on the first image (or the first panoramic image). For example, the box (115) may be an example of an indicator for indicating the location of the currently acquired image (e.g., the first image). For example, the electronic device (101) may display an indicator (117) in a second screen portion (107) of the screen (103) to inform the user of a change in the posture (or movement, direction, position) of the electronic device (101) in order to generate a panoramic image. As an example without limitation, the indicator (117) may indicate the right direction.

[0036] In example (120), the electronic device (101) may acquire a second image through at least one camera. For example, the electronic device (101) may acquire a second image following the first image. By example, without limitation, the second image may be an image immediately following the first image. However, the present disclosure is not limited thereto. For example, the second image may be an image following the first image. In the present disclosure, another specific image immediately following a specific image may represent an image acquired consecutively with the specific image after acquiring the specific image among images acquired through at least one camera of the electronic device (101). In the present disclosure, another specific image following a specific image may represent an image acquired after acquiring the specific image among images acquired through at least one camera of the electronic device (101). In other words, the other specific image following the specific image may represent one of a plurality of images acquired after acquiring the specific image. For convenience of explanation, it is assumed below that the second image is an image immediately following the first image. For example, the electronic device (101) may display a second preview image (121) representing the second image through the first screen portion (105) of the screen (103). For example, the electronic device (101) may display a second thumbnail image (123) representing a panorama image being generated using the second image through the second screen portion (107) of the screen (103). For example, the second thumbnail image (123) may represent a second panorama image in which the first image and the second image are stitched together.For example, the electronic device (101) may display a box (125) indicating the location (or angle of view, area) of the second image in a reference angle of view area defined based on the first image and the second panoramic image. For example, the box (125) may be an example of an indicator for indicating the location of the currently acquired image (e.g., the second image). The box (125) of example (120) may be displayed at a location on the screen (103) that is moved relatively to the upper right compared to the box (115) of example (110). This may indicate that the relative location of the second image with respect to the first image is a location moved to the upper right. In the present disclosure, the relative location of the image may be referenced by the movement of the electronic device (101) (or at least one camera of the electronic device (101)) when acquiring (or capturing) the image. In example (120), while acquiring the second image, the electronic device (101) may recognize (or estimate) that at least one camera has moved to the upper right. For example, the electronic device (101) may display an indicator (127) within the second screen portion (107) of the screen (103) to inform the user of a change in the posture (or movement, direction, position) of the electronic device (101) in order to generate a panoramic image. As an example without limitation, the indicator (127) may indicate the right direction.

[0037] In example (130), the electronic device (101) can acquire a third image through at least one camera. For example, the electronic device (101) can acquire a third image following the second image. For convenience of explanation, it is assumed below that the third image is an image immediately following the second image. For example, the electronic device (101) can display a third preview image (131) representing the third image through the first screen portion (105) of the screen (103). For example, the electronic device (101) can display a third thumbnail image (133) representing a panorama image being generated using the third image through the second screen portion (107) of the screen (103). For example, the third thumbnail image (133) can represent the third panorama image in which the second panorama image (or an image in which the first image and the second image are stitched) and the third panorama image are stitched. For example, the electronic device (101) may display a box (135) indicating the location (or angle of view, area) of the third image in a reference angle of view area defined based on the first image and the third panoramic image. For example, the box (135) may be an example of an indicator for indicating the location of the currently acquired image (e.g., the third image). The box (135) of example (130) may be displayed at a location on the screen (103) that is moved relatively to the upper right compared to the box (125) of example (120). This may indicate that the relative location of the third image with respect to the second image is a location moved to the upper right. For example, the electronic device (101) may display an indicator (137) in a second screen portion (107) of the screen (103) to inform the user of a change in the posture (or movement, direction, position) of the electronic device (101) in order to generate a panoramic image.As an example that is not limited, the indicator (137) can indicate the right direction.

[0038] In example (140), the electronic device (101) can acquire a fourth image through at least one camera. For example, the electronic device (101) can acquire a fourth image following the third image. For convenience of explanation, it is assumed below that the fourth image is an image immediately following the third image. For example, the electronic device (101) can display a fourth preview image (141) representing the fourth image through the first screen portion (105) of the screen (103). For example, the electronic device (101) can display a fourth thumbnail image (143) representing a panorama image being generated using the fourth image through the second screen portion (107) of the screen (103). For example, the fourth thumbnail image (143) can represent the fourth panorama image in which the fourth image is stitched with the third panorama image (or an image in which the first to third images are stitched). For example, the electronic device (101) may display a box (145) indicating the location (or angle of view, area) of the fourth image in a reference angle of view area defined based on the first image and the fourth panoramic image. For example, the box (145) may be an example of an indicator for indicating the location of the currently acquired image (e.g., the fourth image). The box (145) of example (140) may be displayed at a location on the screen (103) that is moved to the lower right relative to the box (135) of example (130). This may indicate that the relative location of the fourth image with respect to the third image is a location moved to the lower right. For example, the electronic device (101) may display an indicator (147) in a second screen portion (107) of the screen (103) to inform the user of a change in the posture (or movement, direction, position) of the electronic device (101) in order to generate a panoramic image.As an example that is not limited, the indicator (147) can indicate the right direction.

[0039] In example (150), the electronic device (101) can acquire a fifth image through at least one camera. For example, the electronic device (101) can acquire a fifth image following the fourth image. For convenience of explanation, it is assumed below that the fifth image is an image immediately following the fourth image. For example, the electronic device (101) can display a fifth preview image (151) representing the fifth image through the first screen portion (105) of the screen (103). For example, the electronic device (101) can display a fifth thumbnail image (153) representing a panorama image being generated using the fifth image through the second screen portion (107) of the screen (103). For example, the fifth thumbnail image (153) can represent the fifth panorama image in which the fourth panorama image (or an image in which the first to fourth images are stitched) and the fifth panorama image are stitched. For example, the electronic device (101) may display a box (155) indicating the location (or angle of view, area) of the fifth image in a reference angle of view area defined based on the first image and the fifth panoramic image. For example, the box (155) may be an example of an indicator for indicating the location of the currently acquired image (e.g., the fifth image). The box (155) of example (150) may be displayed at a location on the screen (103) that is moved to the lower right relative to the box (145) of example (140). This may indicate that the relative location of the fifth image with respect to the fourth image is a location moved to the lower right.

[0040] Referring to example (150), the fifth thumbnail image (153) may include an empty (or, missing) area (153-1) caused in the panoramic image as the position of the currently acquired fifth image is located outside by a specific angle of view in the first direction from the first reference angle of view (e.g., the first reference angle of view (180) in FIG. 1c) defined by the first image. For example, the specific angle of view in the first direction may represent an angle (or position, distance) moved from the first reference angle of view. For example, the area (153-1) may represent a missing part in the fifth panoramic image as the electronic device (101) acquires the fifth image located outside by a specific angle of view in the first direction from the first reference angle of view by moving relatively downward. As an example without limitation, the area (153-1) may be displayed in black. For example, the electronic device (101) may display an indicator (157) within a second screen portion (107) of the screen (103) to notify the user of a change in the posture (or movement, direction, position) of the electronic device (101) in order to generate a panoramic image. As an example without limitation, the indicator (157) may indicate a right direction.

[0041] In example (160), the electronic device (101) can acquire a sixth image through at least one camera. For example, the electronic device (101) can acquire a sixth image following the fifth image. For convenience of explanation, it is assumed below that the sixth image is an image immediately following the fifth image. For example, the electronic device (101) can display a sixth preview image (161) representing the sixth image through the first screen portion (105) of the screen (103). For example, the electronic device (101) can display a sixth thumbnail image (163) representing a panorama image being generated using the sixth image through the second screen portion (107) of the screen (103). For example, the sixth thumbnail image (163) can represent the sixth panorama image in which the fifth panorama image (or an image in which the first to fifth images are stitched) and the sixth panorama image are stitched. For example, the electronic device (101) may display a box (165) indicating the location (or angle of view, area) of the sixth image in a reference angle of view area defined based on the first image and the sixth panoramic image. For example, the box (165) may be an example of an indicator for indicating the location of the currently acquired image (e.g., the sixth image). The box (165) of example (160) may be displayed at a location on the screen (103) that is moved to the lower right relative to the box (155) of example (150). This may indicate that the relative location of the sixth image with respect to the fifth image is a location moved to the lower right.

[0042] Referring to Example (160), the sixth thumbnail image (163) may include an empty (or missing) area (163-1) caused in the panorama image as the position of the currently acquired sixth image is located outside by a different specific angle of view in the first direction from the first reference angle of view defined by the first image. For example, the specific angle of view in the first direction may represent an angle (or position, distance) moved from the first reference angle of view. The difference between the position of the electronic device (101) when acquiring the sixth image in Example (160) and the position of the electronic device (101) when acquiring the first image may be greater than the difference between the position of the electronic device (101) when acquiring the fifth image in Example (150) and the position of the electronic device (101) when acquiring the first image. For example, the size of area (163-1) may be larger than the size of area (153-1). In other words, when the sixth image is acquired, the electronic device (101) may be moved relatively further down than the position of the electronic device (101) when the fifth image is acquired. As a non-limiting example, area (163-1) may be displayed in black. As a non-limiting example, the visual effect (e.g., color) representing the box (165) may be different from the visual effect of the box (155). For example, the box (165) may have a visual effect to indicate that the position of the sixth image is deviated from the first reference angle of view by a threshold angle of view in the first direction. As a non-limiting example, the visual effect of the box (165) may be yellow, and the visual effect of the box (155) (or the boxes (115, 125, 135, 145)) may be white. The visual effect of the box (165) can be used as a notification to the user to generate a natural panoramic image.Additionally, for example, the electronic device (101) may display an additional indicator (165-1) adjacent to the box (165) to inform the user of a change in the posture (or movement, direction, position) of the electronic device (101) in order to create a natural panoramic image. As an example without limitation, the additional indicator (165-1) may indicate an upward direction. For example, the electronic device (101) may display an indicator (167) within a second screen portion (107) of the screen (103) to inform the user of a change in the posture (or movement, direction, position) of the electronic device (101) in order to create a panoramic image. As an example without limitation, the indicator (167) may indicate a right direction.

[0043] According to one embodiment, FIGS. 1a and 1b assume, for convenience of explanation, that the electronic device (101) acquires the first to sixth images, but the present disclosure is not limited thereto. For example, the electronic device (101) may acquire a seventh image after the sixth image. For example, the seventh image may be an image acquired when the electronic device (101) is moved by the preset angle of view (e.g., 180°) from the first image. Hereinafter, it is assumed that the electronic device (101) generates a panoramic image having a smaller angle of view than the preset angle of view by re-identifying the input to the button (109). For example, the panoramic image generated by the electronic device (101) by re-identifying the input to the button (109) may be referred to as a final panoramic image or as a result of panoramic photography. For an example of a final panoramic image, FIG. 1c may be referenced.

[0044] FIG. 1c illustrates an example of a final panoramic image (170) obtained using images obtained within examples (110) to (160) (e.g., the first image to the sixth image). For example, the final panoramic image (170) may correspond to the sixth panoramic image. In other words, the final panoramic image (170) may correspond to the sixth thumbnail image (163) representing the sixth panoramic image.

[0045] According to one embodiment, the final panoramic image (170) may represent a second angle of view (182) in which the first angle of view (181) is removed from the first reference angle of view (180) among the sixth panoramic images. The second angle of view (182) in the first direction of the final panoramic image (170) may be reduced compared to the first reference angle of view (180). This may be because, when acquiring the fifth image and the sixth image, as the electronic device (101) moves relatively downward, a part (e.g., bottom part) of the fifth image and the sixth image located outside the first reference angle of view (180) is removed, and a part (e.g., top part) of the first to fourth images is removed. Accordingly, the final panoramic image (170) may have a second angle of view (182) reduced compared to the first reference angle of view (180) of the initial image (e.g., the first image).

[0046] According to one embodiment, the final panoramic image (170) may include an unnatural area (190) that may occur during stitching between images. For example, the area (190) may include a first image portion (191) and a second image portion (192). For example, the area (190) may include a boundary (193) between the first image portion (191) and the second image portion (192). As an example without limitation, the first image portion (191) may be a portion of a specific image. As an example without limitation, the second image portion (192) may be a portion of another image following the specific image. For example, the boundary (193) may be a line where the edge of the specific image portion and the edge of the other image portion are stitched together. According to one embodiment, the stitching line may include a boundary line where the difference between the images is smallest in at least a portion of the overlapping area of ​​the images, in order to minimize the unnaturalness of the connection portion of the images when the images are stitched (or aligned). According to one embodiment, to ensure the naturalness of the connection portion of the images, blending of the images may be performed by adjusting the weights between the images around the boundary line.

[0047] For example, within the final panoramic image (170), a difference in color between the first image portion (191) and the second image portion (192) may be perceived by the user. Or, for example, within the final panoramic image (170), a boundary (193) may be perceived by the user as being at least partially broken or unnaturally connected. This may occur when identifying the relative position of the other image with respect to the specific image in the area (190) based on motion data. According to one embodiment, the motion data may include sensor values ​​obtained (or stored, recorded) through a motion sensor, such as an accelerometer or gyroscope sensor, when (or while) capturing the image. For example, the motion data may be used to estimate (or identify) the movement of the electronic device (101). If it fails to determine the relative position based on a comparison between the specific image and the other image, the relative position of the other image may be identified using motion data identified when the specific image and the other image are acquired. However, unlike the comparison performed using the feature points of the specific image and the feature points of the other image, an error may occur when identifying the relative position of the other image using the motion data. Accordingly, the final panoramic image (170) may include unnatural parts such as the area (190).

[0048] According to one embodiment, with reference to FIGS. 1a to 1c, an electronic device (101) can perform arrangement and stitching between images when generating a panoramic image using images acquired using at least one camera. For example, the arrangement between images may include identifying (or estimating) the relative position of the current image with respect to the previous image (or the movement of the electronic device (101), the movement of at least one camera of the electronic device (101)), and arranging the current image according to the relative position. For example, the stitching between images may include stitching the previous image (or stitched image) and the current image (or blending, synthesizing, or combining).

[0049] According to one embodiment, regarding the arrangement between images, the electronic device (101) may perform a comparison between images. For example, the electronic device (101) may perform the comparison using feature points of the current image and feature points of the previous image. For example, the electronic device (101) may determine the relative position of the current image with respect to the previous image based on the comparison. However, if the feature points of the current image cannot be identified (or cannot be extracted), the electronic device (101) may determine the relative position of the current image using motion data identified through at least one sensor (e.g., accelerometer, gyroscope) of the electronic device (101) when acquiring the current image. Based on the relative position determined as described above, the electronic device (101) may arrange the current image with respect to the previous image. As described above, the relative position determined using motion data rather than a comparison between images may contain an error.

[0050] According to one embodiment, regarding stitching between images, even if the relative position of the current image is identified and positioned relative to the previous image, since each image represents a three-dimensional external environment in two dimensions, distortion may occur at the boundary between images. To reduce the unnaturalness that may occur at the boundary between images, the electronic device (101) may determine the line with the lowest difference between images as the boundary. However, even if the boundary determined as described above is used, distortion may be perceived due to errors that occur when determining the relative position of the image and differences that occur between images during stitching.

[0051] For example, the distortion may be caused by a failure to identify the parts of the images to be stitched together. In the present disclosure, the electronic device (101) may correct the distortion that may occur while generating a panoramic image using a trained model. A specific example related thereto may be referenced to FIG. 2 below.

[0052] FIG. 2 illustrates an example of a method for generating a corrected panoramic image based on an image generated using a trained model.

[0053] FIG. 2 illustrates examples (160, 210, 220) of a method in which an electronic device (101) generates a corrected panoramic image based on an image generated using a trained model in a panoramic shooting mode. For example (160) of FIG. 2, reference may be made to example (160) of FIG. 1b.

[0054] Referring to FIG. 2, the electronic device (101) may display a screen (103) through a display while acquiring an image to generate a panoramic image. For example, the screen (103) may include a first screen portion (105) in which a preview image representing a part of the external environment included in the acquired image is displayed, a second screen portion (107) in which a thumbnail image representing the panoramic image being generated is displayed, and a button (109) for controlling the capture of the image. The specific details regarding the screen (103) may be substantially the same as those regarding the screen (103) in FIG. 1a and FIG. 1b.

[0055] In the example (210) of FIG. 2, the electronic device (101) can acquire a seventh image through at least one camera. For example, the electronic device (101) can acquire a seventh image following the sixth image. For convenience of explanation, it is assumed below that the seventh image is an image immediately following the sixth image. For example, the electronic device (101) can display a seventh preview image (211) representing the seventh image through the first screen portion (105) of the screen (103). For example, the electronic device (101) can display a seventh thumbnail image (213) representing a panorama image being generated using the seventh image through the second screen portion (107) of the screen (103). For example, the seventh thumbnail image (213) can represent the seventh panorama image in which the seventh image is stitched with the sixth panorama image (or, an image in which the first to sixth images are stitched) and the seventh panorama image are stitched. For example, the electronic device (101) may display a box (215) indicating the location (or angle of view, area) of the seventh image in a reference angle of view area defined based on the first image and the seventh panoramic image. For example, the box (215) may be an example of an indicator for indicating the location of the currently acquired image (e.g., the seventh image). The box (215) of example (210) may be displayed at a location on the screen (103) that is moved relatively to the upper right compared to the box (165) of example (160). This may indicate that the relative location of the seventh image with respect to the sixth image is a location moved to the upper right.

[0056] Referring to Example (210), the seventh thumbnail image (213) may include an empty (or missing) area (213-1) resulting from the panoramic image. For example, the size of the area (213-1) may be equal to or smaller than the size of the area (163-1). In other words, when the seventh image is acquired, the electronic device (101) may be moved in a relatively upward direction compared to the position of the electronic device (101) when the sixth image is acquired. As a non-limiting example, the area (213-1) may be displayed in black. As a non-limiting example, the visual effect (e.g., color) representing the box (215) may be different from the visual effect of the box (165). For example, the visual effect of the box (215) may be the same as the visual effect of the box (155) (or the boxes (115, 125, 135, 145)). For example, the electronic device (101) may display an indicator (217) within a second screen portion (107) of the screen (103) to inform the user of a change in the posture (or movement, direction, position) of the electronic device (101) in order to generate a panoramic image. As an example without limitation, the indicator (217) may indicate a right direction.

[0057] In the example (220) of FIG. 2, the electronic device (101) can acquire an eighth image through at least one camera. For example, the electronic device (101) can acquire an eighth image following the seventh image. For convenience of explanation, it is assumed below that the eighth image is the image immediately following the seventh image. For example, the electronic device (101) can display an eighth preview image (221) representing the eighth image through the first screen portion (105) of the screen (103). For example, the electronic device (101) can display an eighth thumbnail image (223) representing a panorama image being generated using the eighth image through the second screen portion (107) of the screen (103). For example, the eighth thumbnail image (223) can represent the eighth panorama image in which the eighth image is stitched with the seventh panorama image (or, an image in which the first to seventh images are stitched) and the eighth panorama image are stitched. For example, the electronic device (101) may display a box (225) indicating the location (or angle of view, area) of the eighth image in a reference angle of view area defined based on the first image and the eighth panoramic image. For example, the box (215) may be an example of an indicator for indicating the location of the currently acquired image (e.g., the seventh image). The box (225) of example (220) may be displayed at a location on the screen (103) that is moved relatively to the right compared to the box (215) of example (210). This may indicate that the relative location of the eighth image with respect to the seventh image is a location moved to the right. For example, the electronic device (101) may display an indicator (227) in a second screen portion (107) of the screen (103) to inform the user of a change in the posture (or movement, direction, position) of the electronic device (101) in order to generate a panoramic image.As an example that is not limited, the indicator (227) can indicate the right direction.

[0058] According to one embodiment, with reference to examples (210) and (220), the electronic device (101) may determine a region (213-1) as a generation region. In a non-limiting example, the electronic device (101) may determine a field of view (or a portion of an image used to generate the seventh panoramic image, e.g., the sixth image) containing a region (213-1) that may be recognized as a distortion within the seventh panoramic image being generated as a generation region. In the above example, the region (213-1) is shown as being determined as a generation region after the seventh panoramic image is generated, but the present disclosure is not limited thereto. For example, the electronic device (101) may determine a region (163-1) as a generation region before generating the seventh panoramic image. For example, the electronic device (101) may provide the generation region to a trained model (250). For example, the trained model (250) may be stored within the electronic device (101) (or in the memory of the electronic device (101)). As an example without limitation, the trained model (250) may be a generative artificial intelligence model. In the above example, it is described that a generative area is provided to the trained model (250), but the present disclosure is not limited thereto. For example, the electronic device (101) may provide the trained model (250) with text information describing the generative area along with the generative area. For example, the electronic device (101) may use the trained model (250) to generate an image corresponding to the generative area. Generating an image corresponding to the generative area using the trained model (250) may be performed while the electronic device (101) generates a panoramic image.

[0059] For example, the eighth thumbnail image (223) may be an image from which distortion, such as the region (213-1) of the seventh thumbnail image (213), has been removed. In other words, the eighth panoramic image may be an image updated based on an image generated using a trained model (250). For example, the eighth panoramic image, in which the seventh panoramic image and the eighth image are stitched, may be an image in which the distorted region (213-1) within the seventh panoramic image has been removed and an image generated using the trained model (250) has been inserted. According to one embodiment, the electronic device (101) may obtain an image generated using the trained model (250) by providing information about a generated region including the region (213-1) to the trained model (250). By example, without limitation, the information about the generated region may include image (or video) information and information for describing the image. By example, without limitation, the information about the generated region may include a prompt describing the generated region. Accordingly, distortion can be removed from the eighth thumbnail image (223) representing the eighth panoramic image. Although not illustrated in FIG. 2, if the shooting of the panoramic image is stopped in the example (220), the electronic device (101) can generate the eighth panoramic image as a final panoramic image. In other words, compared to the final panoramic image (170) of FIG. 1c, the final panoramic image obtained in FIG. 2 may be an image with distortion removed and a wider field of view.

[0060] Referring to the above description, the electronic device (101) may use a trained model (250) to eliminate distortion that may occur while generating a panoramic image. Accordingly, the electronic device (101) according to the present disclosure may eliminate a reduction in the field of view of the panoramic image or unnatural areas (or distortions) within the panoramic image.

[0061] In the present disclosure, the electronic device (101) may acquire images through at least one camera, arrange the images, and stitch the images while generating a panoramic image. The operations performed by the electronic device (101) to generate a panoramic image may be referenced below with reference to FIG. 4.

[0062] The electronic device (101) can perform identification of parts of acquired images to be stitched while generating a panoramic image. The electronic device (101) can determine (or estimate) the relative position of the images using motion data, or determine that the identification has failed if the difference between parts of the images to be stitched exceeds a threshold value, or if the size of the area of ​​the stitched image located outside the reference angle of view exceeds a reference size. For example, an example of the failure of the identification caused by determining the relative position of the images using motion data may be referenced in FIGS. 5 to 6b below. For example, an example of the failure of the identification caused by the difference between parts of the images to be stitched exceeding a threshold value may be referenced in FIGS. 7 to 8b below. For example, an example of the failure of the identification caused by the size of the area of ​​the stitched image located outside the reference angle of view exceeding a reference size may be referenced in FIGS. 9 to 10b below. If the above identification fails, the electronic device (101) may determine the portion of the panoramic image for which the identification failed as a generation area. The electronic device (101) may generate an image corresponding to the generation area using a trained model (250). For generating an image corresponding to the generation area using the trained model (250), reference may be made to FIGS. 11 to 12c below.

[0063] Figure 3 is an exemplary schematic view of an electronic device.

[0064] FIG. 3 illustrates a schematic diagram of the components included in an electronic device (101). The electronic device (101) of FIG. 3 may be an example of the electronic device (101) of FIG. 1a through 1c and the electronic device (101) of FIG. 2. For example, the electronic device (101) may be an example of the electronic device (1401) of FIG. 14. For example, the electronic device (101) may include at least a part of the electronic device (1401) of FIG. 14.

[0065] According to one embodiment, the electronic device (101) may include at least one processor (310), at least one camera (320), at least one sensor (330), a display (340), and a memory (350). For example, at least one processor (310), at least one camera (320), at least one sensor (330), a display (340), and a memory (350) may be electrically and / or operably coupled with each other by a communication bus. Hereinafter, the operably coupled hardware components may mean that a direct connection or an indirect connection between the hardware components is established via a wire or wirelessly so that a second hardware component is controlled by a first hardware component among the hardware components. Although illustrated based on different blocks, the embodiment is not limited thereto, and some of the hardware components illustrated in FIG. 3 (e.g., at least one processor (310), communication circuit (340), or memory (350)) may be included in a single integrated circuit such as a system on a chip (SoC) or a system in package (SIP). The type and / or number of hardware components included in the electronic device (101) are not limited to those illustrated in FIG. 3. For example, the electronic device (101) may include only some of the hardware components illustrated in FIG. 3.

[0066] For example, the electronic device (101) may be implemented in various form factors. For example, the electronic device (101) may include not only an electronic device including a bar-type display, but also an electronic device including a display that is a flexible display. For example, the flexible display may include an electronic device including a foldable display, an electronic device including a multi-foldable display, or an electronic device including a rollable display. Additionally, for example, the electronic device (101) may include a tablet PC. Additionally, for example, the electronic device (101) may be implemented as a wearable device. For example, the wearable device may include a watch-shaped device. However, the present disclosure is not limited thereto.

[0067] According to one embodiment, at least one processor (310) of an electronic device (101) may include a hardware component for processing communication and / or data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), and a field programmable gate array (FPGA). As an example, the hardware component for processing data may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing unit (DSP), a microcontroller (MCU), and / or a neural processing unit (NPU). For example, at least one processor (310) may have the structure of a multi-core processor such as a dual core, a quad core, or a hexa core. The at least one processor (310) of FIG. 3 may have substantially the same properties as the processor (1420) of FIG. 14.

[0068] For example, at least one processor (310) may include various processing circuits and / or multiple processors. For example, the term “processor” as used herein, including in the claims, may include various processing circuits including at least one processor, and one or more of said at least one processor may be configured to perform the various functions described below in a distributed manner, individually and / or collectively. As used below, where “processor,” “at least one processor,” and “one or more processors” are described as being configured to perform various functions, these terms encompass, for example, but not limited to, situations where one processor performs some of the cited functions and other processor(s) perform other parts of the cited functions, and also situations where one processor can perform all of the cited functions. Additionally, said at least one processor may include a combination of processors that perform the enumerated / disclosed various functions, for example, in a distributed manner. At least one processor may execute program instructions to achieve or perform the various functions.

[0069] According to one embodiment, at least one camera (320) of the electronic device (101) may include one camera or a plurality of cameras. For example, at least one camera (320) may be used to acquire images of the actual environment (or external environment). For example, the electronic device (101) may acquire images of the actual environment by driving at least one camera (320). For example, the electronic device (101) may execute a camera application and acquire images within the panorama shot mode of the executed camera application. For example, the electronic device (101) may generate a panorama image using the acquired images. As an example without limitation, the images may represent a preset field of view of the actual environment. For example, the preset field of view may be 180°. However, the present disclosure is not limited thereto. For example, the electronic device (101) may acquire images showing a field of view smaller than the preset field of view according to user input regarding the screen of the camera application, and generate a panoramic image using the acquired images. For example, at least one camera (320) may be an example of the camera module (1480) of FIG. 14. At least one camera (320) may include at least a part of the camera module (1480) of FIG. 14.

[0070] According to one embodiment, at least one sensor (330) of the electronic device (101) may include a motion sensor for sensing the posture (or movement, position, or direction) of the electronic device (101). For example, the motion sensor may include an accelerometer or a gyroscope. The electronic device (101) may identify the posture of the electronic device (101) when acquiring an image through at least one camera (320) using motion data acquired through at least one sensor (330). For example, based on the motion data, the electronic device (101) may identify whether it has moved by a preset angle of view while acquiring images through at least one camera (320). Or, for example, based on the motion data, the electronic device (101) may identify the relative position of the specific image with respect to another image immediately preceding the specific image when acquiring the specific image through at least one camera (320). For example, the relative position may include the direction and / or distance in which the specific image is moved from the other image. For example, the relative position may be referenced by the movement of the electronic device (101) (or at least one camera (320) of the electronic device (101)) when acquiring (or capturing) the specific image. For example, the relative position may be used to position (or align) the specific image with respect to the other image. For example, at least one sensor (330) may be an example of the sensor module (1476) of FIG. 14. At least one sensor (330) may include at least a part of the sensor module (1476) of FIG. 14.

[0071] According to one embodiment, a display (340) of an electronic device (101) may be used to display a screen. For example, the display (340) may include a display panel that includes a display area for displaying a screen. For example, the screen may include a screen of a camera application (or a user interface) (e.g., the screen (103) of FIG. 1a). It may include a preview image (e.g., the preview image (111) of FIG. 1a) representing the actual environment in which the electronic device (101) is shooting (or acquiring an image) and a thumbnail image (e.g., the thumbnail image (113) of FIG. 1a) representing the panorama image being generated. For example, the screen may include a first part in which the preview image is displayed (e.g., the first screen part (105) of FIG. 1a) and a second part in which the thumbnail image is displayed (e.g., the second screen part (107) of FIG. 1a). Additionally, the screen may include a button of the camera application (e.g., button (109) of FIG. 1a) for starting (or initiating) the capture of an image, stopping (or ceasing) the capture of an image, resuming the capture of an image, or stopping (or ending) the capture of an image. For example, the display (340) may be an example of the display module (1460) of FIG. 14. The display (340) may include at least a part of the display module (1460) of FIG. 14.

[0072] According to one embodiment, the electronic device (101) may include a memory (350). The memory (350) may include a hardware component for storing data and / or instructions that are input to or / or output from at least one processor (310). The memory (350) may include, for example, a volatile memory such as random-access memory (RAM), and / or a non-volatile memory such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disk, and embedded multimedia card (eMMC). For example, memory (350) may be an example of memory (1430) of FIG. 14. Memory (350) may include at least a portion of memory (1430) of FIG. 14.

[0073] According to one embodiment, within the memory (350) of the electronic device (101), one or more instructions (or commands) representing operations and / or operations to be performed on data by at least one processor (310) of the electronic device (101) may be stored. A set of one or more instructions may be referred to as a program, firmware, operating system, process, routine, sub-routine, and / or application. Hereinafter, the statement that an application is installed within the electronic device (e.g., electronic device (101)) may mean that one or more instructions provided in the form of an application are stored within the memory (350), and that said one or more applications are stored in an executable format (e.g., a file having an extension specified by the operating system of the electronic device (101)) by the processor of the electronic device. According to one embodiment, the electronic device (101) may perform operations by executing one or more instructions stored in the memory (350). For example, the above one or more instructions may cause at least some of the operations of the electronic device (101) to be performed when executed by at least one processor (310).

[0074] According to one embodiment, the memory (350) may include (or store) a trained model (250). For example, the trained model (250) may be used to generate an image corresponding to a generation area determined based on the failure to identify parts of the images to be stitched. For example, the image generated by the trained model (250) and corresponding to the generation area may be referred to as an image area, an image part, a panoramic image part, or a part of the panoramic image's field of view. For example, the trained model (250) may generate the image corresponding to the generation area by acquiring information for stitching the images and using the acquired information as input. Specific details regarding the method of generating the image corresponding to the generation area using the trained model (250) may be referenced below in FIGS. 11, FIGS. 12a, and FIGS. 12c.

[0075] In FIG. 3, an example of an on-device where a trained model (250) is stored in memory (350) is illustrated, but the present disclosure is not limited thereto. For example, the trained model (250) may be stored in an external electronic device (e.g., a server) other than the electronic device (101). The electronic device (101) may request the external electronic device to generate an image corresponding to a generated area.

[0076] For example, information for stitching images may include the generated area and text information describing the generated area. By example, without limitation, the generated area may include a part (or a part of the field of view) of the stitched image. By example, without limitation, the text information describing the generated area may include at least one word representing at least one object included in the generated area, or at least one sentence describing a scene of the generated area. For example, the text information may be referred to as a prompt. By example, without limitation, the text information may be generated from another trained model that uses the generated area as input. For example, the other trained model may be referred to as a prompt generation model. The prompt generation model may be included in the prompt generation module (1270) of FIG. 12b. Specific details regarding the method of generating the text information using the prompt generation module (1270) may be referenced below in FIG. 11 and FIG. 12b.

[0077] According to one embodiment, although not illustrated in FIG. 3, the electronic device (101) may include a plurality of modules for performing real-time image stitching. By example, without limitation, the plurality of modules may be implemented in software within at least one processor (310) and / or memory (350). By example, without limitation, the plurality of modules may be implemented in software and hardware. For example, the plurality of modules may include an image placement module, an image stitching module, and an image generation module.

[0078] According to one embodiment, the image placement module may be used to identify (or estimate) the relative position of an image (or the movement of the electronic device (101) or the movement of the at least one camera (320)) by using information (e.g., image, motion data) obtained through at least one camera (320) and / or at least one sensor (330). By example, without limitation, the image placement module may include an image estimation module and a sensor estimation module. For example, the image estimation module may identify feature points of the images and use the identified feature points to identify (or estimate) the relative position of the image (or the movement of the electronic device (101) or the movement of the at least one camera (320)). For example, the sensor estimation module may identify (or estimate) the relative position of the image (or the movement of the electronic device (101) or the movement of the at least one camera (320)) by using motion data identified through at least one sensor (330). For example, the image placement module can place images according to identified relative positions. For example, if the image placement module cannot identify the relative positions of images through comparison between images and identifies the relative positions of images using motion data, it may request the image generation module to generate images.

[0079] According to one embodiment, the image stitching module may identify (or search for) the boundaries of the placed images and stitch the images along the boundaries. By example, without limitation, the image stitching module may include a boundary search module, a result generation module, and a thumbnail generation module. For example, the boundary search module may be used to identify boundaries from the placed images. For example, the boundary may represent a line where the sum of the difference values ​​of the color data (or pixel values) of the images (or within the area where the images overlap) is minimized. For example, the result generation module may stitch the images along the boundaries by adjusting the weights (or stitching weights) of the images. For example, the result generation module may update the stitched image using an image obtained from the image generation module. For example, when the result generation module has obtained all the images for generating a panorama image, it may generate a final panorama image and store the final panorama image in memory (350). For example, the thumbnail generation module may be used to generate a thumbnail image to be displayed through the display (340). For example, the image stitching module may request the image generation module to generate an image if the value of the boundary exceeds a threshold value. Additionally, for example, the image stitching module may request the image generation module to generate an image if the size of the area where the stitched image is not located within the reference angle of view exceeds a reference size.

[0080] According to one embodiment, the image generation module may be used to generate a requested image using a trained model (250). For example, the image generation module may generate an image corresponding to a generation area. For example, the image generation module may include a generation area setting module, a prompt generation module (e.g., the prompt generation module (1270) of FIG. 12b), and a trained model (250). For example, the generation area setting module may be used to set a generation area (or a reference area, a masked reference area). For example, the generation area may be determined based on the failure to identify parts of the images to be stitched. For example, the prompt generation module may be used to generate text information (e.g., a prompt) among the information for stitching images to be input to the trained model (250). For example, the trained model (250) may be a generative artificial intelligence model.

[0081] Hereinafter, in the present disclosure, the case in which the trained model (250) is stored in the electronic device (101) (or memory (350)) is exemplified, but the present disclosure is not limited thereto. For example, the electronic device (101) may request an external electronic device (e.g., a server) to generate an image and receive an image generated from the trained model within the external electronic device. Alternatively, for example, the electronic device (101) may perform image generation for a thumbnail image using the trained model (250) within the electronic device (101) and request the trained model within the external electronic device to generate an image for a final panorama image.

[0082] Figure 4 illustrates an example of the operation flow for generating a panoramic image.

[0083] At least some of the above methods of FIG. 4 may be performed by the electronic device (101) of FIG. 3. For example, at least some of the above methods may be controlled by at least one processor (310) of the electronic device (101). In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0084] According to one embodiment, in operation (400), the electronic device (101) can acquire an image. For example, the electronic device (101) can drive at least one camera (320). For example, the electronic device (101) can acquire an image through at least one camera (320).

[0085] Although not illustrated in FIG. 4, the electronic device (101) may execute a camera application prior to operation (400). For example, the electronic device (101) may activate a panoramic shooting mode of the executed camera application. For example, the electronic device (101) may identify an input for generating a panoramic image within the panoramic shooting mode. By example, without limitation, the input may be an input to a button (e.g., button (109) in FIG. 1a) on the screen of the camera application (e.g., screen (103) in FIG. 1a). For example, the electronic device (101) may acquire images through at least one camera (320) to generate the panoramic image. In FIG. 4, it is illustrated that one image is acquired, but the present disclosure is not limited thereto. For example, the electronic device (101) may acquire images sequentially to generate the panoramic image.

[0086] Although not illustrated in FIG. 4, the electronic device (101) may drive at least one sensor (330) when acquiring an image in operation (400). For example, the electronic device (101) may identify motion data while acquiring an image by using at least one sensor (330). For example, the motion data may be used to identify the posture (or movement, direction, position) of the electronic device (101) when acquiring an image.

[0087] According to one embodiment, in operation (405), the electronic device (101) can identify whether shooting has been stopped. As an example without limitation, the electronic device (101) can identify whether acquiring an image has been stopped after identifying the input to a button (e.g., button (109) in FIG. 1a) on a screen of a camera application (e.g., screen (103) in FIG. 1a), depending on whether the input to the button on the screen is identified again. For example, the electronic device (101) can identify that shooting has been stopped if the input to the button on the screen is identified again. Alternatively, the electronic device (101) can identify that shooting has not been stopped (or that shooting is being maintained) if the input to the button on the screen is not identified again.

[0088] In operation (405), if the electronic device (101) identifies that the shooting has stopped, it may perform operation (425). Alternatively, in operation (405), if the electronic device (101) identifies that the shooting has not stopped, it may perform operation (410).

[0089] According to one embodiment, in operation (410), the electronic device (101) can identify whether the image has been captured for the maximum angle of view for a panoramic image. According to one embodiment, the operation of identifying whether the image has been captured for the maximum angle of view for a panoramic image may include the operation of identifying (or confirming) whether the image being captured has been captured for the maximum length. For example, the electronic device (101) can identify the location where the first image among the images for generating the panoramic image was acquired. For example, the location where the first image was acquired may be identified based on motion data identified while acquiring the first image. For example, the electronic device (101) can identify whether the image has been captured for the maximum angle of view for a panoramic image by identifying whether an image has been acquired at a location moved from the location by the maximum angle of view. In other words, the electronic device (101) can identify that the images for generating the panoramic image have been captured to the maximum angle of view for the panoramic image if the images for generating the panoramic image include an image having a position moved by the maximum angle of view (or, among the images, the final image is an image having a position moved by the maximum angle of view). Alternatively, the electronic device (101) can identify that the images for generating the panoramic image have not been captured to the maximum angle of view for the panoramic image if the images for generating the panoramic image do not include an image having a position moved by the maximum angle of view (or, among the images, the final image is an image having a position moved by an angle smaller than the maximum angle of view). The maximum angle of view may be a preset angle of view for the panoramic image (e.g., 180°).

[0090] According to one embodiment, in operation (410), if the electronic device (101) identifies that the image has been captured to the maximum angle of view for a panoramic image, it may perform operation (425). Alternatively, in operation (410), if the electronic device (101) identifies that the image has not been captured to the maximum angle of view for a panoramic image, it may perform operation (415).

[0091] For convenience of explanation, the electronic device (101) is assumed to have acquired a first image, which is the first image among the images for generating a panoramic image, and acquired a second image following the first image. For example, when the electronic device (101) acquires the first image, the first image, the operation (415) and the operation (420) may be omitted.

[0092] According to one embodiment, in operation (415), the electronic device (101) may perform image placement. For example, the electronic device (101) may perform image placement for the second image. For example, the image placement may include determining the relative position of the second image with respect to the first image and placing the second image according to the determined relative position. Specific details regarding operation (415) may be referenced below in FIGS. 5 to 6b.

[0093] According to one embodiment, in operation (420), the electronic device (101) may perform image stitching. For example, the electronic device (101) may perform stitching of the placed second image and the first image. For example, the image stitching may represent stitching a portion of the second image and a portion of the first image. For example, the image in which the portion of the second image and the portion of the first image are stitched may be referred to as a stacked image, or a panoramic image prior to the final panoramic image (e.g., the seventh panoramic image in the example (210) of FIG. 2). For example, the electronic device (101) may generate a panoramic image prior to the final panoramic image (or an intermediate panoramic image) by performing image stitching. For example, the intermediate panoramic image may be used to generate the final panoramic image. In one example, the final panoramic image may be generated as the intermediate panoramic image is updated. Specific details regarding the operation (420) may be referenced in FIGS. 7 to FIGS. 10b below.

[0094] According to one embodiment, after operation (420), the electronic device (101) may perform operation (400) again. For example, the electronic device (101) may acquire a third image following the second image. For example, after acquiring the third image, the electronic device (101) may perform operation (415) again by performing operation (405) and operation (410). In the operation (415) performed again, the electronic device (101) may perform image placement with respect to the second image and the third image. For example, the electronic device (101) may determine the relative position of the third image with respect to the second image and place the third image according to the relative position. Additionally, in the operation (420) performed again, the electronic device (101) may perform image stitching with the third image and an image in which the portion of the first image and the portion of the second image are stitched together.

[0095] Although not illustrated in FIG. 4, the electronic device (101) may generate a preview image and display the preview image through the display (340) while performing operation (400) to operation (420). For example, the preview image may be changed (or updated) whenever an image is acquired. Although not illustrated in FIG. 4, the electronic device (101) may generate a thumbnail image and display the thumbnail image through the display (340) while performing operation (400) to operation (420). For example, the thumbnail image may be changed (or updated) whenever a stitched image is generated.

[0096] According to one embodiment, in operation (425), the electronic device (101) may generate a panoramic image. The panoramic image generated in operation (425) may represent a final panoramic image. For example, the electronic device (101) may generate a final panoramic image and perform processing on the generated final panoramic image. In one example, the final panoramic image may be generated according to an update to a panoramic image (or intermediate panoramic image) generated prior to the final panoramic image. In an example, without limitation, the processing may include compression and / or storage of the final panoramic image. In other words, after generating the final panoramic image, the electronic device (101) may compress the final panoramic image to store it in memory (350) and store it in memory (350). In an example, without limitation, the electronic device (101) may provide (or display) the stored final panoramic image to a user by running an application (e.g., an album application, a gallery application) for providing the stored image.

[0097] Although not illustrated in FIG. 4, the electronic device (101) may generate an image representing at least a portion of a panoramic image using a trained model (250) before performing operation (425). For example, the image representing at least a portion may correspond to a generated area. For example, the generated area may be determined based on the failure to identify portions of images to be stitched during operation (415) and / or operation (420). As an example without limitation, the electronic device (101) may determine (or set) the generated area while performing operation (400) to operation (420) (or while generating a panoramic image) and generate an image corresponding to the determined generated area using the trained model (250). In other words, the electronic device (101) may perform the operations of FIG. 11 while performing operation (400) to operation (420) (or while generating a panoramic image).

[0098] As a non-limiting example, the electronic device (101) may postpone the execution of operation (425) when the maximum angle of view for the panoramic image in operation (410) has been captured, but the operations of FIG. 11 for acquiring the image using the trained model (250) are being performed. In other words, the electronic device (101) may perform operation (425) using the generated image after generating an image from the trained model (250) in FIG. 11. Accordingly, in operation (425), the electronic device (101) may generate a corrected final panoramic image.

[0099] FIG. 5 illustrates an example of a flow of operations for determining a generation area while performing the arrangement of images to generate a panoramic image.

[0100] At least some of the above methods of FIG. 5 may be performed by the electronic device (101) of FIG. 3. For example, at least some of the above methods may be controlled by at least one processor (310) of the electronic device (101). In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0101] For convenience of explanation, it is assumed that the electronic device (101) acquires a first image, which is the first image among the images for generating a panoramic image, and acquires a second image following the first image.

[0102] According to one embodiment, in operation (500), the electronic device (101) may perform a comparison of images. For example, the electronic device (101) may perform the comparison using feature points of the first image and feature points of the second images. For example, the electronic device (101) may identify (or extract) feature points of the first image. For example, the electronic device (101) may identify (or extract) feature points of the second image. As an example without limitation, identifying (or extracting) feature points of the image may be performed when the image is acquired. Or, as an example without limitation, identifying (or extracting) feature points of the image may be performed when the comparison is performed in operation (500). For example, feature points of the image may be extracted from content (or objects) within the image. As an example that is not limited, feature points of an image can define the appearance of content (or objects).

[0103] In operation (500), the electronic device (101) can identify, by performing the comparison, the difference value between the feature points of the first image and the feature points of the second image, the number of the feature points of the second image, or whether the content within the second image includes repeated patterns. For example, the electronic device (101) can identify whether the relative position of the second image with respect to the first image can be determined as a result of the comparison. For example, the electronic device (101) can identify that the relative position of the second image with respect to the first image cannot be determined. An example of the case where the relative position of the second image cannot be determined may be referenced below in FIG. 6a.

[0104] FIG. 6a illustrates examples of cases where the relative position of an image cannot be determined.

[0105] FIG. 6a illustrates examples of images (600, 605) in which the relative position cannot be determined. For example, each of the images (600, 605) may be an example of the second image.

[0106] For example, the image (600) may include content having repeated patterns. As a non-limiting example, the content included in the image (600) may include windows. For example, the electronic device (101) may identify (or extract) feature points (601) from the content included in the image (600). As a non-limiting example, the feature points (601) identified from the windows of the image (600) may have repeated patterns. For example, the electronic device (101) may identify that the image (600) includes content formed by repeated patterns based on the arrangement of the feature points (601). When the image (600) including content formed by repeated patterns is acquired after another image having substantially the same repeated patterns, the electronic device (101) may not be able to identify the relative position of the image (600) with respect to the other image because the pattern of the feature points to be compared is generally similar. Accordingly, the electronic device (101) can identify that the relative position of the image (600) cannot be determined when the image (600) includes repeated patterns.

[0107] For example, the image (605) may contain content that is relatively lacking in texture. As a non-limiting example, the content included in the image (605) may include a wall. For example, the electronic device (101) may identify (or extract) feature points (606) from the content included in the image (605). As a non-limiting example, the number of feature points (606) identified from the wall of the image (605) may be four. For example, the electronic device (101) may identify that the number of feature points (606) of the image (605) is less than a reference number (e.g., five). For example, when the electronic device (101) identifies the number of feature points (606) that is less than the reference number, it may not be able to identify the relative position of the image (605) with respect to another image acquired prior to the image (605) because there is a lack of feature points to compare. Accordingly, the electronic device (101) can identify that the relative position of the image (605) cannot be determined if the number of feature points (606) of the image (605) is less than the reference number. In the above example, the reference number is used only as a criterion for identifying content for which it is difficult to extract feature points from the content within the image, and the present disclosure is not limited thereto. For example, the electronic device (101) may identify content for which it is difficult to extract feature points from the content within the image without using the reference number.

[0108] Referring again to FIG. 5, the electronic device (101) can identify the number of feature points of the second image by performing the comparison. For example, the electronic device (101) can identify whether the number of feature points of the second image is less than a reference number by performing the comparison. For example, the reference number can be used to determine whether feature points identified (or extracted) from the content within the image can be used for the comparison. For example, the electronic device (101) can recognize that the comparison cannot be performed using feature points identified from the content within the second image, depending on the number of feature points of the second image being less than the reference number. For example, the electronic device (101) can identify that the result of the comparison cannot determine the relative position of the second image, depending on the number of feature points of the second image being less than the reference number.

[0109] Alternatively, for example, the electronic device (101) may identify whether the content in the second image contains repeated patterns by performing the comparison. For example, the electronic device (101) may identify the result of the comparison as being unable to determine the relative position of the second image depending on the content in the second image containing repeated patterns.

[0110] In the example of FIG. 6a, the number of feature points identified from an image containing content formed by repeated patterns is exemplified as being less than the reference number, but the present disclosure is not limited thereto. For example, the electronic device (101) can identify a difference value between the feature points of the first image and the feature points of the second image by performing the comparison. For example, the electronic device (101) can identify whether the difference value exceeds a reference value by performing the comparison. The reference value may be used to identify the degree to which the second image has changed from the first image. In other words, the electronic device (101) can recognize that the electronic device (101) moved relatively quickly (or that the change in movement of the electronic device (101) is relatively large) while acquiring the second image after acquiring the first image, depending on the difference value that exceeds the reference value. For example, the electronic device (101) can identify that the result of the comparison cannot determine the relative position of the second image according to the difference value exceeding the reference value.

[0111] According to one embodiment, in operation (505), the electronic device (101) can identify whether the relative position of the image can be determined using the result of the comparison. For example, the electronic device (101) can identify whether the relative position of the second image with respect to the first image can be determined using the result of the comparison between the first image and the second image. In one example, the electronic device (101) may identify the movement of the electronic device (101) while acquiring the first image and the second image according to the result of the comparison between the first image and the second image.

[0112] In operation (505), the electronic device (101) can perform operation (510) if it cannot determine the relative position of the image. Alternatively, in operation (505), the electronic device (101) can perform operation (520) if it can determine the relative position of the image.

[0113] According to one embodiment, in operation (510), the electronic device (101) can determine the relative position of an image using motion data of the electronic device (101). For example, the electronic device (101) can determine the relative position of the second image using the motion data identified while acquiring the first image and the second image. For example, the motion data can be identified through at least one sensor (330) while acquiring the first image and the second image. For example, the electronic device (101) can identify a change from the posture (or position, direction) of the electronic device (101) when the first image is acquired to the posture (or position, direction) of the electronic device (101) when the second image is acquired using the motion data identified while acquiring the first image and the second image. For example, the electronic device (101) can determine the relative position of the second image according to the identified change. In one example, the electronic device (101) may identify the movement of the electronic device (101) while acquiring the first image and the second image according to the motion data identified while acquiring the first image and the second image.

[0114] According to one embodiment, in operation (515), the electronic device (101) may store a generation area. For example, the generation area may be determined using the motion data identified during the acquisition of the first image and the second image, the first image, and the second image. For example, the generation area may be an area including a location (or field of view) of the first image within the panoramic image according to the motion data identified during the acquisition of the first image and the second image, and a location (or field of view) of the second image within the panoramic image according to the motion data identified during the acquisition of the first image and the second image. For example, the generation area may include (or correspond to) a portion (or field of view) within the panoramic image to be stitched the first image and the second image. For example, the generation area may be included in information for stitching the first image and the second image.

[0115] According to one embodiment, after performing operations (510) and (515), in operation (530), the electronic device (101) may place an image according to a relative position. For example, the electronic device (101) may place the second image according to the relative position of the second image with respect to the first image determined using motion data in operation (510). The second image placed with respect to the first image may be stitched with the first image according to the operations of FIG. 7. When a relative position determined based on motion data is used, there is a higher possibility that the second image may be inaccurately placed with respect to the first image. Accordingly, the portion of the second image to be stitched with the portion of the first image may cause distortion. For example, the distortion may be displayed within the thumbnail image, such as the area (213-1) shown in the example (210) of FIG. 2. The above distortion can be removed according to the operation (525) to be described later.

[0116] For convenience of explanation, it is assumed that the electronic device (101) acquires a third image following the second image in accordance with the operation (400) of FIG. 4. For example, after acquiring the third image, the electronic device (101) may perform a comparison between the second image and the third image in operation (500). In operation (505), the electronic device (101) may identify whether the relative position of the third image can be determined using the result of the comparison between the second image and the third image. For example, the electronic device (101) may determine the relative position of the third image by identifying feature points of the second image and feature points of the third image. In other words, the electronic device (101) may identify the result of the comparison between the second image and the third image as being able to determine the relative position of the third image with respect to the second image. Accordingly, the electronic device (101) can perform the operation (520).

[0117] According to one embodiment, in operation (520), the electronic device (101) can identify whether a generated area is stored. For example, the electronic device (101) can identify whether a generated area is stored in memory (350).

[0118] In operation (520), the electronic device (101) can perform operation (525) if the generated area is stored. Alternatively, in operation (520), the electronic device (101) can perform operation (530) if the generated area is not stored.

[0119] For convenience of explanation, in the following, it is assumed that the generated area is stored in the operation (515) performed in relation to the second image. For example, the electronic device (101) can identify that the generated area is stored in the operation (520) and perform the operation (525).

[0120] According to one embodiment, in operation (525), the electronic device (101) may request image generation. For example, the electronic device (101) may obtain information for stitching the first image and the second image and provide said information to a trained model (250). For example, the information for stitching the first image and the second image may include the generation area stored in memory (350). Additionally, for example, the information for stitching the first image and the second image may include text information describing the generation area. For example, the text information may include at least one word representing at least one object included in the generation area, or at least one sentence describing a scene in the generation area. For example, the text information may be referenced as a prompt. As an example without limitation, the text information may be generated from another trained model that uses the generation area as input. For example, the other trained model may be referenced as a prompt generation model. The above prompt generation model may be included in the prompt generation module (1270) of FIG. 12b.

[0121] The image generation request performed in operation (525) can be referred to as an inpainting generation request, as the generation area is a part of the area located between the acquired images.

[0122] Although not illustrated in FIG. 5, as operation (525) is performed, the electronic device (101) can remove the generated area stored in memory (350). For example, when requesting image generation, the electronic device (101) can remove the generated area from memory (350).

[0123] The electronic device (101) may perform operation (530) after requesting image generation according to operation (525). For example, the electronic device (101) may place the third image according to a relative position in operation (530). For example, the electronic device (101) may place the third image according to the relative position of the third image with respect to the second image determined by performing the comparison in operation (500) and operation (505). The third image placed with respect to the second image may be stitched with the second image (or an image in which the first image and the second image are stitched) according to the operations of FIG. 7. Since a relative position determined based on feature points is used, the portion of the third image to be stitched with the portion of the second image may contain relatively less distortion (or not contain any distortion).

[0124] In the above example, it is assumed that the generated area is stored according to the operations (500), (510), and (515) performed in relation to the second image, but the present disclosure is not limited thereto. For example, the electronic device (101) may identify that, after performing a comparison between the first image and the second image in operation (500), it can determine the relative position of the second image with respect to the first image using the result of the comparison in operation (505). In this case, the electronic device (101) may identify in operation (520) that the generated area is not stored and perform operation (530). For example, the electronic device (101) may place the second image according to the relative position in operation (530). For example, the electronic device (101) may position the second image according to the relative position of the second image with respect to the first image determined by performing the comparison in operation (500) and operation (505). The second image positioned with respect to the first image may be stitched with the first image according to the operations of FIG. 7. Since the relative position determined based on feature points is used, the portion of the second image to be stitched with the portion of the first image may contain (or not contain) relatively less distortion. At this time, the electronic device (101) may bypass the operation (525) requesting image generation. For example, the electronic device (101) may bypass (or refrain from) providing information to the trained model (250) for stitching the first image and the second image.

[0125] The operations of the electronic device (101) illustrated in FIG. 5 can be exemplified as in FIG. 6b over time.

[0126] FIG. 6b illustrates an example of a method for identifying regions to be generated using a trained model while performing the arrangement of images.

[0127] FIG. 6b illustrates an example (610) in which an electronic device (101) acquires a second image (612) following a first image (611), an example (620) in which it acquires a third image (621) following the second image (612), and an example (630) in which it acquires a fourth image (631) following the third image (621).

[0128] In example (610), the electronic device (101) can acquire a first image (611) and, according to motion data identified while acquiring the first image (611), place the first image (611) at a location within the panoramic image to be generated (e.g., the initial angle of view (0)).

[0129] Although not illustrated in FIG. 6b, the electronic device (101) can display a preview image representing the first image (611) and a thumbnail image representing the first image (611) through the display (340).

[0130] In example (610), the electronic device (101) can acquire a second image (612). For example, the electronic device (101) can compare the second image (612) with the first image (611). The electronic device (101) can identify that, as a result of comparing the second image (612) with the first image (611), the relative position (615) of the second image (612) with respect to the first image (611) can be determined. For example, as in the operation (510) of FIG. 5, the electronic device (101) can determine the relative position of the second image (612) with respect to the first image (611) using motion data. The electronic device (101) can determine the relative position (615) of the second image (612) and then position the second image (612) relative to the first image (611) according to the relative position (615).

[0131] Although not illustrated in FIG. 6b, the electronic device (101) can stitch a second image (612) and a first image (611) arranged according to relative positions (615). Additionally, the electronic device (101) can display a preview image representing the second image (612) through a display (340). Additionally, the electronic device (101) can display a thumbnail image through a display (340) representing an image in which the first image (611) and the second image (612) are stitched (or a panoramic image prior to the final panoramic image).

[0132] In example (620), the electronic device (101) can acquire a third image (621). For example, the electronic device (101) can compare the third image (621) with the second image (612). The electronic device (101) can identify that, as a result of comparing the third image (621) with the second image (612), the relative position of the third image (621) with respect to the second image (612) cannot be determined. Accordingly, the electronic device (101) can determine the relative position of the third image (621) using motion data identified while acquiring the second image (612) and the third image (621). For example, the electronic device (101) can determine the relative position of the third image (621) using motion data and then position the third image (621) relative to the second image (612) according to the relative position.

[0133] Although not illustrated in FIG. 6b, the electronic device (101) can stitch a third image (621) and a second image (612) (or an image in which the first image (611) and the second image (612) are stitched) according to relative positions determined based on motion data. At this time, distortion may occur in a portion of the second image (612) to be stitched (or an image in which the first image (611) and the second image (612) are stitched) and a portion of the third image (613) to be stitched. For example, when relative positions determined based on motion data are used, there is a higher possibility that the third image (621) will be inaccurately positioned with respect to the second image (612). Accordingly, a portion of the second image (612) and a portion of the third image (613) to be stitched may cause distortion. Additionally, the electronic device (101) may display a preview image representing the third image (621) through the display (340). Additionally, the electronic device (101) may display a thumbnail image through the display (340) representing the stitched image of the third image (621) and the second image (612) (or the stitched image of the first image (611) and the second image (612)) (or the panoramic image prior to the final panoramic image).

[0134] In example (630), the electronic device (101) can acquire a fourth image (631). For example, the electronic device (101) can compare the fourth image (631) with the third image (621). The electronic device (101) can identify that, as a result of comparing the fourth image (631) with the third image (621), the relative position of the fourth image (631) with respect to the third image (621) can be determined. After determining the relative position of the fourth image (631), the electronic device (101) can position the fourth image (631) with respect to the third image (621) according to the relative position.

[0135] In example (630), the electronic device (101) may perform an image generation request for a generation area (639). For example, the generation area (639) may be determined (or set) when determining the relative position of the third image (621) using motion data in example (620). In an example without limitation, the generation area (639) may include (or correspond to) the position (or angle of view, area) of the second image (612) and the third image (621) within the panoramic image to be generated. In an example without limitation, the generation area (639) may include (or correspond to) the position (or angle of view, area) of the second image (612), the third image (621), and the fourth image (631) within the panoramic image to be generated. Alternatively, as a non-limiting example, the generated area (639) may include (or correspond to) a location (or field of view, area) of a part of the second image (612), a third image (621), and a part of the fourth image (631). According to one embodiment, if motion estimation for the generated area (639) is successful, the generated area (e.g., an area that failed for image analysis) may be reset. For example, the generated area (639) may include a location (or field of view, area) of the third image (621) whose relative position is determined using motion data, and may include a location (or field of view, area) extending from the third image (621).

[0136] Although not illustrated in FIG. 6b, the electronic device (101) can stitch a fourth image (631) and a third image (621) (or an image in which the first image (611) to the third image (621) are stitched) arranged according to relative positions determined based on feature points. Additionally, the electronic device (101) can display a preview image representing the fourth image (631) through the display (340). Additionally, the electronic device (101) can display a thumbnail image through the display (340) representing an image in which the fourth image (631) and the third image (621) (or an image in which the first image (611) to the third image (621) are stitched) are stitched (or a panoramic image prior to the final panoramic image). Alternatively, the electronic device (101) can generate an image corresponding to the generated area (639) using a trained model (250). The electronic device (101) can replace (or update, change) the generated area (639) in the image in which the fourth image (631) and the third image (621) are stitched with an image generated using the trained model (250). As described above, replacing the generated area (639) with an image generated using the trained model (250) is a request to generate an area for a part located between the acquired images, and can be referred to as an inpainting generation request. Accordingly, the electronic device (101) can display a thumbnail image in which the generated area (639) has been replaced (or updated, changed) with the generated image through the display (340).

[0137] FIG. 7 illustrates an example of a flow of operations for determining a generation area while performing stitching of images to generate a panoramic image.

[0138] At least some of the above methods of FIG. 7 may be performed by the electronic device (101) of FIG. 3. For example, at least some of the above methods may be controlled by at least one processor (310) of the electronic device (101). In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0139] For convenience of explanation, it is assumed that the electronic device (101) acquires a first image, which is the first image among the images for generating a panoramic image, and acquires a second image following the first image.

[0140] According to one embodiment, before performing the operation (700), the electronic device (101) may determine the relative position of the second image with respect to the first image according to the operations of FIG. 5, and position the second image according to the determined relative position. As an example without limitation, the relative position of the second image may be determined based on feature points by performing a comparison between the first image and the second image. Or, as an example without limitation, the relative position of the second image may be determined based on motion data identified while acquiring the first image and the second image.

[0141] According to one embodiment, in operation (700), the electronic device (101) can identify the boundaries of the images. For example, the electronic device (101) can identify an area where the second image overlaps the first image according to the relative position of the second image. The overlapping area may also be referred to as an area where the first image overlaps the second image. For example, when a part of the first image is stitched with a part of the second image, the edge of the part of the first image and the edge of the part of the second image may be located (or included) within the overlapping area. For example, the edge of the part of the first image and the edge of the part of the second image may be defined by the boundary.

[0142] For example, the electronic device (101) can identify the boundary within the overlapping area. As a non-limiting example, the electronic device (101) can identify the difference between the edge of the part of the first image and the edge of the part of the second image by using first color data representing the edge of the part of the first image and second color data representing the edge of the part of the second image. For example, the electronic device (101) can identify the boundary using the difference. An example of identifying the boundary using the difference may be referenced below in FIG. 8a.

[0143] FIG. 8a illustrates an example of a method for identifying the boundary between parts of images to be stitched together.

[0144] FIG. 8a illustrates an example of a method for identifying the boundary between the parts to be stitched of the first image (801) and the second image (802). FIG. 8a illustrates an example of identifying the boundary between the first image (801) and the second image (802) for convenience of explanation, but the present disclosure is not limited thereto. For example, the second image (802) may be an example of an image currently acquired through at least one camera (320), and the first image (801) may be an example of an image (or a panoramic image prior to the final panoramic image) in which images acquired prior to the second image (802) are stitched together.

[0145] Referring to FIG. 8a, the first image (801) and the second image (802) may overlap each other. For example, depending on the relative position of the second image (802) with respect to the first image (801), at least a portion of the second image (802) may overlap with at least a portion of the first image (801). For example, the first image (801) and the second image (802) may overlap in an area (803).

[0146] For example, the electronic device (101) can identify the difference between the first color data of the first image (801) and the second color data of the second image (802) within the area (803). For example, the first color data may include color information of each pixel of the first image (801) for representing the first image (801). Also, for example, the second color data may include color information of each pixel of the second image (802) for representing the second image (802). As a non-limiting example, the color information may include RGB data. However, the present disclosure is not limited thereto. For example, the color information may include color space data. For the difference between the first color data and the second color data for each pixel within the area (803), the example (804-1) may be referenced.

[0147] Referring to Example (804-1), the difference between the first color data and the second color data for each pixel within the area (803) is illustrated. For example, the difference between the first color data and the second color data for each pixel within the area (803) can be calculated as a specific value (e.g., 3, 5, 6, 10, 11, 12, 20, 30, 40, 50). For example, the specific value may represent the difference between a specific pixel value of the first image (801) and a specific pixel value of the second image (802). For example, the electronic device (101) can identify a boundary (805) within the area (803) along a first direction (d1) where the sum of the differences between the first color data and the second color data for the pixels within the area (803) is minimized. As a non-limiting example, the value of the boundary (805) may be 30 (=10+5+5+5+5). As described above, in the present disclosure, a graph-cut algorithm may be used as a method for identifying the boundary (805). However, the present disclosure is not limited thereto.

[0148] In Example (804-1), for convenience of explanation, a boundary (805) formed in a straight line along a first direction (d1) in some pixels (e.g., a 5x5 pixel array) of area (803) is shown, but the present disclosure is not limited thereto. For example, within area (803), the boundary (805) may be formed at least partially diagonally along the first direction (d1) and the second direction (d2). For a specific example related thereto, refer to Example (804-2).

[0149] Referring to example (804-2), the electronic device (101) can identify a boundary (807) by using the difference between the first color data and the second color data for each of the pixels within the region (803). For example, the boundary (807) may represent a boundary for all pixels of the region (803), including the boundary (805) of example (804-1). For example, the boundary (807) may define the edge of the portion (801-1) of the first image (801) to be stitched with the second image (802) and the edge of the portion (802-1) of the second image (802) to be stitched with the first image (801). For example, a boundary (807) can be identified using the difference between the first color data of the edge of the portion (801-1) of the first image (801) and the second color data of the edge of the portion (802-1) of the second image (802). In other words, the value of the boundary (807) can have a minimum value for the difference (or sum of differences) between the first color data of the first image (801) and the second color data of the second image (802) within the area (803).

[0150] Referring again to FIG. 7, in operation (705), the electronic device (101) can identify whether the difference between edges defined by a boundary exceeds a threshold value. For example, the electronic device (101) can identify whether the difference between the edge of the portion of the first image and the edge of the portion of the second image exceeds the threshold value. In other words, the electronic device (101) can identify whether the value of the boundary (e.g., boundary (807) in FIG. 8a) defining the edge of the portion of the first image and the edge of the portion of the second image exceeds the threshold value.

[0151] For example, the threshold value can be used to determine whether a boundary having the most similar color data within a region (e.g., boundary (807) in FIG. 8a) is an unnatural boundary, even if the first image and the second image form an overlapping region according to the arrangement. In other words, the electronic device (101) can recognize that there is a large error between the images depending on the value of the boundary having a value exceeding the threshold value.

[0152] According to one embodiment, in operation (705), the electronic device (101) may perform operation (710) when the difference between the edges defined by the boundary exceeds the threshold value. Alternatively, in operation (705), the electronic device (101) may perform operation (715) when the difference between the edges defined by the boundary is less than or equal to the threshold value.

[0153] According to one embodiment, in operation (710), the electronic device (101) may store a generation area. For example, the generation area may be determined using the motion data identified during the acquisition of the first image and the second image, the first image, and the second image. For example, the generation area may be an area including the boundary between the edge of the portion of the first image and the edge of the portion of the second image according to the motion data identified during the acquisition of the first image and the second image. For example, the generation area including the boundary may include (or correspond to) a portion (or field of view) within the panoramic image to which the first image and the second image are to be stitched. For example, the generation area may be included in information for stitching the first image and the second image.

[0154] According to one embodiment, in operation (715), the electronic device (101) can identify whether the image includes at least a portion of the boundary. For example, the electronic device (101) can identify whether the second image includes at least a portion of the boundary that is included in the generated area. In the example, the second image may include the boundary because it includes the edge of the portion of the second image. Accordingly, the electronic device (101) can perform operation (725).

[0155] In the operation (725) performed after the operation (715), the electronic device (101) can perform stitching of images. For example, the electronic device (101) can perform stitching of the first image and the second image. For example, the electronic device (101) can stitch the portion of the first image (e.g., portion (801-1) of FIG. 8a) and the portion of the second image (e.g., portion (802-1) of FIG. 8a). For example, the image in which the first image and the second image are stitched may be an image stitched along the boundary between the edge of the portion of the first image and the edge of the portion of the second image.

[0156] The image generation request performed in operation (725) can be referred to as an inpainting generation request, as the generation area is a part of the area located between the acquired images.

[0157] For convenience of explanation, it is assumed below that the electronic device (101) acquires a third image following the second image according to the operation (400) of FIG. 4. For example, after acquiring the third image, the electronic device (101) can identify the boundary between the second image (or, an image in which the first image and the second image are stitched, a panoramic image prior to the final panoramic image) and the third image in operation (700). In operation (705), the electronic device (101) can identify whether the difference between the edges defined by the boundary between the second image and the third image exceeds the threshold value. For convenience of explanation, it is assumed below that the difference between the edges defined by the boundary between the second image and the third image is less than or equal to the threshold value. In operation (715), the electronic device (101) can identify whether the third image includes at least a portion of the boundary between the first image and the second image. For example, the boundary of the operation (715) may be the boundary between the first image and the second image included in the generated area stored in the operation (710). Hereinafter, it is assumed that the third image does not include at least a part of the boundary between the first image and the second image.

[0158] According to one embodiment, in operation (720), the electronic device (101) may request image generation. For example, the electronic device (101) may obtain information for stitching the first image and the second image and provide said information to a trained model (250). For example, the information for stitching the first image and the second image may include the generation area stored in memory (350). Additionally, for example, the information for stitching the first image and the second image may include text information describing the generation area. For example, the text information may include at least one word representing at least one object included in the generation area, or at least one sentence describing a scene in the generation area. For example, the text information may be referenced as a prompt. As an example without limitation, the text information may be generated from another trained model that uses the generation area as input. For example, the other trained model may be referenced as a prompt generation model. The above prompt generation model may be included in the prompt generation module (1270) of FIG. 12b.

[0159] After performing operation (720), in operation (725), the electronic device (101) can perform stitching of images. For example, the electronic device (101) can perform stitching of the second image (or, an image in which the first image and the second image are stitched, a panoramic image prior to the final panoramic image) and the third image. For example, the electronic device (101) can stitch a portion of the second image and a portion of the third image. For example, the image in which the second image and the third image are stitched may be an image stitched along the boundary between the edge of the portion of the second image and the edge of the portion of the third image.

[0160] Although not illustrated in FIG. 7, the electronic device (101) may update (or replace, change) the thumbnail image displayed through the display (340) using the stitched image after performing the operation (725). For example, while displaying a thumbnail image representing the first image, if the first image and the second image are stitched, the electronic device (101) may update the thumbnail image to the image in which the first image and the second image are stitched. Additionally, for example, while displaying a thumbnail image representing the image in which the first image and the second image are stitched, the electronic device (101) may update the thumbnail image to the image in which the second image (or the image in which the first image and the second image are stitched) and the third image are stitched.

[0161] The operations of the electronic device (101) illustrated in FIG. 7 can be exemplified as in FIG. 8b over time.

[0162] FIG. 8b illustrates an example of a method for identifying regions to be generated using a trained model while performing stitching of images.

[0163] FIG. 8b illustrates an example (810) in which an electronic device (101) acquires a second image (812) following a first image (811), an example (820) in which it acquires a third image (821) following the second image (812), an example (830) in which it acquires a fourth image (831) following the third image (821), and an example (840) in which it acquires a fifth image (841) following the fourth image (831).

[0164] In example (810), the electronic device (101) can acquire a first image (811) and, according to motion data identified while acquiring the first image (811), place the first image (811) at a location within the panoramic image to be generated (e.g., the initial angle of view (0)).

[0165] Although not illustrated in FIG. 8b, the electronic device (101) can display a preview image representing the first image (811) and a thumbnail image representing the first image (811) through the display (340).

[0166] In example (810), the electronic device (101) can acquire a second image (812). For example, the electronic device (101) can determine the relative position of the second image (812) with respect to the first image (811) and then position the second image (812) with respect to the first image (811) according to the relative position. For example, the electronic device (101) can identify a boundary (817) within the area where the first image (811) and the second image (812) overlap. For example, the boundary (817) can be identified using the difference between the first color data of the first image (811) and the second color data of the second image (812) within the area where the first image (811) and the second image (812) overlap. In example (810), the electronic device (101) can identify that the value of the boundary (817) is below a threshold value. Accordingly, the electronic device (101) may bypass (or refrain from) storing the generated area including the boundary (817) in memory (350).

[0167] Although not illustrated in FIG. 8b, the electronic device (101) can stitch the second image (812) and the first image (811). Additionally, the electronic device (101) can display a preview image representing the second image (812) through the display (340). Additionally, the electronic device (101) can display a thumbnail image through the display (340) representing the image in which the first image (811) and the second image (812) are stitched (or the panoramic image prior to the final panoramic image).

[0168] In example (820), the electronic device (101) can acquire a third image (821). For example, the electronic device (101) can determine the relative position of the third image (821) with respect to the second image (812) and then position the third image (821) relative to the second image (812) according to the relative position. For example, the electronic device (101) can identify a boundary (827) within the area where the second image (812) (or an image where the first image (811) and the second image (812) are stitched) and the third image (821) overlap. For example, the boundary (827) can be identified using the difference between the second color data of the second image (812) and the third color data of the third image (821) within the area where the second image (812) and the third image (821) overlap. In example (820), the electronic device (101) can identify that the value of the boundary (827) exceeds a threshold value. Accordingly, the electronic device (101) can store a generated area (829) containing the boundary (827) in memory (350).

[0169] Although not illustrated in FIG. 8b, the electronic device (101) can stitch the third image (821) and the second image (812) (or an image in which the first image (811) and the second image (812) are stitched). Additionally, the electronic device (101) can display a preview image representing the third image (821) through the display (340). Additionally, the electronic device (101) can display a thumbnail image through the display (340) representing the second image (812) (or an image in which the first image (811) and the second image (812) are stitched) and the third image (821) stitched (or a panoramic image prior to the final panoramic image).

[0170] In example (830), the electronic device (101) can acquire a fourth image (831). For example, the electronic device (101) can determine the relative position of the fourth image (831) with respect to the third image (821) and then position the fourth image (831) with respect to the third image (821) according to the relative position. In example (830), the electronic device (101) can identify whether the fourth image (831) includes at least a portion of the boundary (827) of the generation area (829) stored in memory (350). In example (830), the electronic device (101) can identify that the fourth image (831) includes a portion (827-1) of the boundary (827). Accordingly, the electronic device (101) can bypass (or refrain from) the image generation request.

[0171] Although not illustrated in FIG. 8b, the electronic device (101) can stitch the fourth image (831) and the third image (821) (or an image in which the first image (811) to the third image (821) are stitched). Additionally, the electronic device (101) can display a preview image representing the fourth image (831) through the display (340). Additionally, the electronic device (101) can display a thumbnail image through the display (340) representing the third image (821) (or an image in which the first image (811) to the third image (821) are stitched) and the image in which the fourth image (831) is stitched (or a panoramic image prior to the final panoramic image).

[0172] In example (840), the electronic device (101) can acquire a fifth image (841). For example, the electronic device (101) can determine the relative position of the fifth image (841) with respect to the fourth image (831) and then position the fifth image (841) with respect to the fourth image (831) according to the relative position. In example (840), the electronic device (101) can identify whether the fifth image (841) includes at least a portion of the boundary (827) of the generated area (829) stored in memory (350). In example (840), the electronic device (101) can identify that the fifth image (841) does not include at least a portion of the boundary (827). Accordingly, the electronic device (101) can perform an image generation request.

[0173] In example (830), the electronic device (101) may perform an image generation request for a generation area (829). For example, the generation area (829) may be determined (or set) when the value of the boundary (827) in example (820) exceeds the threshold value. In a non-limiting example, the generation area (829) may include (or correspond to) the location (or angle of view, area) of the boundary (827) between the second image (812) and the third image (821) within the panoramic image to be generated. In a non-limiting example, the generation area (829) may include (or correspond to) the location (or angle of view, area) of the second image (812) and the third image (821) within the panoramic image to be generated. Alternatively, as a non-limiting example, the generated area (829) may include (or correspond to) a location (or angle of view, area) of a part of the second image (812) and a part of the third image (821). In other words, the generated area (829) may include a location (or angle of view, area) of the boundary (827) and a location (or angle of view, area) extending from the boundary (827).

[0174] Although not illustrated in FIG. 8b, the electronic device (101) can stitch the fifth image (841) and the fourth image (831) (or an image in which the first image (811) to the fourth image (831) are stitched). Additionally, the electronic device (101) can display a preview image representing the fifth image (841) through the display (340). Additionally, the electronic device (101) can display a thumbnail image through the display (340) representing the fourth image (831) (or an image in which the first image (811) to the fourth image (831) are stitched) and the image in which the fifth image (841) is stitched (or a panoramic image prior to the final panoramic image). Alternatively, the electronic device (101) can generate an image corresponding to the generation area (829) using a trained model (250). The electronic device (101) can replace (or update, change) the generated region (829) in the image in which the fifth image (841) and the fourth image (831) are stitched with an image generated using a trained model (250). Accordingly, the electronic device (101) can display a thumbnail image in which the generated region (829) has been replaced (or updated, changed) with the generated image through a display (340).

[0175] FIG. 9 illustrates an example of a flow of operations for determining a generation area in a stitched image to generate a panoramic image.

[0176] At least some of the above methods of FIG. 9 may be performed by the electronic device (101) of FIG. 3. For example, at least some of the above methods may be controlled by at least one processor (310) of the electronic device (101). In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0177] For convenience of explanation, it is assumed that the electronic device (101) acquires a first image, which is the first image among the images for generating a panoramic image, and acquires a second image following the first image.

[0178] According to one embodiment, before performing the operation (900), the electronic device (101) may determine the relative position of the second image with respect to the first image according to the operations of FIG. 5, and may place the second image according to the determined relative position. Additionally, the electronic device (101) may stitch the second image and the first image according to the operations of FIG. 7. As a non-limiting example, a portion of the first image (e.g., portion (801-1) of FIG. 8a) and a portion of the second image (e.g., portion (802-1) of FIG. 8a) may be stitched. Hereinafter, the image in which the first image and the second image are stitched may be referred to as the first stitched image.

[0179] According to one embodiment, in operation (900), the electronic device (101) can identify a first reference angle of view of the initial image. For example, the electronic device (101) can identify the first reference angle of view defined by the first image. For example, the first reference angle of view may represent an angle of view in a first direction (e.g., vertical direction or vertical direction) defined by the first image.

[0180] According to one embodiment, in operation (905), the electronic device (101) can identify a second reference angle of view of the stitched image. For example, the electronic device (101) can identify the second reference angle of view defined by the first stitched image. For example, the second reference angle of view may be defined by the first stitched image and may represent an angle of view in a second direction (e.g., horizontal or side direction) that is perpendicular to the first direction.

[0181] According to one embodiment, in operation (910), the electronic device (101) can identify whether the size of an area where the stitched image is not located within the reference angle of view area exceeds the reference size. For example, the area where the stitched image is not located within the reference angle of view area can be exemplified as the area (1009) of FIG. 10a described later. For example, the electronic device (101) can identify the reference angle of view area defined by the first reference angle of view and the second reference angle of view. For example, the electronic device (101) can identify the area where the first stitched image is not located within the reference angle of view area and identify the size of the area. For example, the electronic device (101) can identify whether the size of the area exceeds the reference size.

[0182] In relation to operations (900) to operations (910), examples of the first reference angle of view, the second reference angle of view, the reference angle of view area, and the area within the reference angle of view area where the stitched image is not located may be referenced in FIG. 10a below.

[0183] FIG. 10a illustrates an example of a method for identifying a reference field of view region for a stitched image.

[0184] FIG. 10a illustrates an example of a method for identifying a reference field of view area defined with respect to a stitched image. The stitched image (1003) of FIG. 10a may be an image in which a first image (1001) and a second image (1002) are stitched together. For example, the first image (1001) may be the initial image. For example, the second image (1002) may be the image following the first image (1001).

[0185] For example, the electronic device (101) can identify a first reference angle of view (1005) in a first direction defined by a first image (1001). For example, the first reference angle of view (1005) may represent the angle of view of the first image (1001) in the first direction. For example, the electronic device (101) can identify a second reference angle of view (1007) in a second direction defined by a stitched image (1003). For example, the second reference angle of view (1007) may represent the angle of view of the stitched image (1003) in the second direction.

[0186] For example, the electronic device (101) can identify a reference angle of view area (1008) defined by a first reference angle of view (1005) and a second reference angle of view (1007). For example, the reference angle of view area (1008) may be defined by the first reference angle of view (1005) in the first direction and by the second reference angle of view (1007) in the second direction. For example, the reference angle of view area (1008) may be included in the entire angle of view area of ​​the final panoramic image to be generated.

[0187] For example, the electronic device (101) can identify an area (1009) where the stitched image (1003) is not located within the reference angle of view area (1008). For example, the electronic device (101) can identify the area (1009) as a portion within the reference angle of view area (1008) where the reference angle of view area (1008) and the stitched image (1003) do not overlap. For example, the electronic device (101) can identify the size of the area (1009). For example, the electronic device (101) can recognize that if the size of the area (1009) exceeds the reference size, the loss of the angle of view of the final panoramic image will be caused by the area (1009).

[0188] Referring again to FIG. 9, in operation (910), the electronic device (101) can perform operation (915) if the size of the area where the stitched image is not located within the reference angle of view exceeds the reference size. Alternatively, in operation (910), the electronic device (101) can perform operation (920) if the size of the area where the stitched image is not located within the reference angle of view is less than or equal to the reference size.

[0189] According to one embodiment, in operation (915), the electronic device (101) may request image generation. For example, the electronic device (101) may obtain information for stitching the first image and the second image and provide said information to a trained model (250). For example, the information for stitching the first image and the second image may include a generation area. For example, the generation area may include an area within the reference viewing angle area where the stitched image is not located. In this case, the area included in the generation area may have a size exceeding the reference size. Additionally, for example, the information for stitching the first image and the second image may include text information describing the generation area. For example, the text information may include at least one word representing at least one object included in the generation area, or at least one sentence describing the scene of the generation area. For example, the text information may be referenced as a prompt. As a non-limiting example, the text information may be generated from another trained model that uses the generation area as input. For example, the other trained model may be referred to as a prompt generation model. The prompt generation model may be included in the prompt generation module (1270) of FIG. 12b.

[0190] The image generation request performed in operation (915) can be referred to as an outpainting generation request, as the generation area is a part of the area located outside the acquired images.

[0191] According to one embodiment, in operation (920), the electronic device (101) may perform the generation of a panoramic image. Operation (920) may be substantially the same as operation (425) of FIG. 4. FIG. 9 is illustrated as performing operation (920) after performing operation (910) and operation (915), but the present disclosure is not limited thereto. For example, the electronic device (101) may perform an update of a thumbnail image after performing operation (910) and operation (915), and repeat operation of FIG. 4 again for an image obtained again in operation (400) of FIG. 4 (e.g., a third image following the second image).

[0192] The operations of the electronic device (101) illustrated in FIG. 9 can be exemplified as in FIG. 10b over time.

[0193] FIG. 10b illustrates an example of a method for identifying regions to be generated using a trained model in a stitched image.

[0194] FIG. 10b illustrates an example (1010) in which an electronic device (101) acquires a second image (1012) following a first image (1011), an example (1020) in which it acquires a third image (1021) following the second image (1012), and an example (1030) in which it acquires a fourth image (1031) following the third image (1021).

[0195] In example (1010), the electronic device (101) can acquire a first image (1011) and, according to motion data identified while acquiring the first image (1011), place the first image (1011) at a location (e.g., initial angle of view (0)) within the panoramic image to be generated.

[0196] Although not illustrated in FIG. 10b, the electronic device (101) can display a preview image representing the first image (1011) and a thumbnail image representing the first image (1011) through the display (340).

[0197] In example (1010), the electronic device (101) can acquire a second image (1012). For example, the electronic device (101) can determine the relative position of the second image (1012) with respect to the first image (1011), and then position the second image (1012) relative to the first image (1011) according to the relative position. For example, the electronic device (101) can identify a boundary within the area where the first image (1011) and the second image (1012) overlap. For example, the electronic device (101) can stitch the first image (1011) and the second image (1012) along said boundary. Accordingly, a first stitched image (1013) can be generated.

[0198] In example (1010), the electronic device (101) can identify a first reference angle of view (1015) defined by a first image (1011). For example, the electronic device (101) can identify a second reference angle of view (1017) defined by a first stitched image (1013). For example, the electronic device (101) can identify a reference angle of view area (1018) defined by the first reference angle of view (1015) and the second reference angle of view (1017). For example, the reference angle of view area (1018) can be defined with respect to the first stitched image (1013). For example, the electronic device (101) can identify an area (1019) within the reference angle of view area (1018) where the first stitched image (1013) is not located. For example, the electronic device (101) can identify that the size of the area (1019) is less than or equal to the reference size. Accordingly, the electronic device (101) can bypass (or refrain from) the image generation request.

[0199] Although not illustrated in FIG. 10b, the electronic device (101) may display a preview image representing a second image (1012) through the display (340). Additionally, the electronic device (101) may display a thumbnail image representing a first stitched image (1013) through the display (340).

[0200] In example (1020), the electronic device (101) can acquire a third image (1021). For example, the electronic device (101) can determine the relative position of the third image (1021) with respect to the second image (1012) and then position the third image (1021) relative to the second image (1012) according to the relative position. For example, the electronic device (101) can identify a boundary within the area where the second image (1012) (or the first stitched image (1013)) and the third image (1021) overlap. For example, the electronic device (101) can stitch the second image (1012) (or the first stitched image (1013)) and the third image (1021) along said boundary. Accordingly, a second stitched image (1023) can be created.

[0201] In example (1020), the electronic device (101) can identify a second reference angle of view (1027) defined by a second stitched image (1023). For example, the electronic device (101) can identify a reference angle of view area (1028) defined by a first reference angle of view (1015) and a second reference angle of view (1027). For example, the reference angle of view area (1028) can be defined with respect to the second stitched image (1023). For example, the electronic device (101) can identify an area (1029) within the reference angle of view area (1028) where the second stitched image (1023) is not located. For example, the electronic device (101) can identify that the size of the area (1029) is less than or equal to the reference size. Accordingly, the electronic device (101) can bypass (or refrain from) the image generation request.

[0202] Although not illustrated in FIG. 10b, the electronic device (101) may display a preview image representing a third image (1021) through the display (340). Additionally, the electronic device (101) may display a thumbnail image representing a second stitched image (1023) through the display (340).

[0203] In example (1030), the electronic device (101) can acquire a fourth image (1031). For example, the electronic device (101) can determine the relative position of the fourth image (1031) with respect to the third image (1021) and then position the fourth image (1031) relative to the third image (1021) according to the relative position. For example, the electronic device (101) can identify a boundary within the area where the third image (1021) (or the second stitched image (1023)) and the fourth image (1031) overlap. For example, the electronic device (101) can stitch the third image (1021) (or the second stitched image (1023)) and the fourth image (1031) along said boundary. Accordingly, a third stitched image (1033) can be created.

[0204] In example (1030), the electronic device (101) can identify a second reference angle of view (1037) defined by a third stitched image (1033). For example, the electronic device (101) can identify a reference angle of view area (1038) defined by a first reference angle of view (1015) and a second reference angle of view (1037). For example, the reference angle of view area (1038) can be defined with respect to the third stitched image (1033). For example, the electronic device (101) can identify an area (1039) within the reference angle of view area (1038) where the third stitched image (1033) is not located. For example, the electronic device (101) can identify that the size of the area (1039) exceeds the reference size. Accordingly, the electronic device (101) can perform an image generation request.

[0205] In example (1030), the electronic device (101) may perform an image generation request for a generation area. For example, the generation area may be determined (or set) when the size of the area (1039) in example (1030) exceeds the reference size. In an example without limitation, the generation area may include (or correspond to) the location (or angle of view, area) of the area (1039) within the panoramic image to be generated. In an example without limitation, the generation area may include (or correspond to) the location (or angle of view, area) of an image (e.g., a fourth image (1031)) included in the second reference angle of view in the second direction of the area (1039) within the panoramic image to be generated. In other words, the generation area may include the location (or angle of view, area) of the area (1039) and may include a location (or angle of view, area) extending from the area (1039).

[0206] Although not illustrated in FIG. 10b, the electronic device (101) may display a preview image representing the fourth image (1031) through the display (340). Additionally, the electronic device (101) may display a thumbnail image representing the third stitched image (1033) through the display (340). Alternatively, the electronic device (101) may generate an image corresponding to a generated area including the region (1039) using a trained model (250). The electronic device (101) may replace (or update, change) the generated area among the third stitched images (1033) with an image generated using the trained model (250). As described above, replacing the generated area including the region (1039) with an image generated using the trained model (250) is a request to generate a region for a part located outside the acquired images, and may be referred to as an outpainting generation request. Accordingly, the electronic device (101) can display a thumbnail image through the display (340) in which the generated area has been replaced (or updated, changed) with the generated image.

[0207] FIG. 11 illustrates an example of the operation flow for a method of generating an image corresponding to a generated area using a trained model.

[0208] At least some of the above methods of FIG. 11 may be performed by the electronic device (101) of FIG. 3. For example, at least some of the above methods may be controlled by at least one processor (310) of the electronic device (101). In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0209] According to one embodiment, in operation (1100), the electronic device (101) can obtain information for stitching images.

[0210] For example, the electronic device (101) may obtain the information including a generation area. As a non-limiting example, the generation area of ​​the information may include a generation area stored in operation (515) (e.g., the generation area (639) of FIG. 6b). Or, as a non-limiting example, the generation area of ​​the information may include a generation area stored in operation (710) (e.g., the generation area (829) of FIG. 8b). Or, as a non-limiting example, the generation area of ​​the information may include an area having a size exceeding the reference size exemplified in operation (915) (e.g., the area (1039) of FIG. 10b).

[0211] According to one embodiment, the information may include text information describing the generation area. For example, the text information may include at least one word representing at least one object included in the generation area, or at least one sentence describing a scene of the generation area. For example, the text information may be referred to as a prompt. By example, without limitation, the text information may be generated from another trained model that uses the generation area as input. For example, the other trained model may be referred to as a prompt generation model. The prompt generation model may be included in the prompt generation module (1270) of FIG. 12b.

[0212] According to one embodiment, in operation (1105), the electronic device (101) may provide the information to the trained model (250). For example, the electronic device (101) may provide the information obtained in operation (1100) to the trained model (250) stored in memory (350). Alternatively, for example, the electronic device (101) may provide the information obtained in operation (1100) to the trained model stored in an external electronic device (e.g., a server).

[0213] The operation (1100) and the operation (1105) may include requesting the generation of an image of the operation (525) of FIG. 5, requesting the generation of an image of the operation (720) of FIG. 7, and requesting the generation of an image of the operation (915) of FIG. 9.

[0214] According to one embodiment, in operation (1110), the electronic device (101) can acquire a portion of images stitched together. For example, the electronic device (101) can acquire a portion of images stitched together (or a stitched image) using a trained model (250). For example, the portion of images stitched together (or a stitched image) may be an image corresponding to a generated area. For example, the portion of images stitched together (or a stitched image) may be at least a part of the final panoramic image to be generated.

[0215] According to one embodiment, in operation (1115), the electronic device (101) can update a panoramic image and a thumbnail image. For example, the electronic device (101) can replace a portion of the panoramic image being generated with a portion of the images that are stitched together (or, a stitched image). For example, the electronic device (101) can update the thumbnail image to represent the panoramic image that is replaced with the portion of the images that are stitched together (or, a stitched image).

[0216] The operations of the electronic device (101) illustrated in FIG. 11 can be exemplified as in FIG. 12a over time.

[0217] FIG. 12a illustrates an example of a method for generating an image corresponding to a generated region using a trained model.

[0218] FIG. 12a illustrates an example of a method in which an electronic device (101) uses a trained model (250) to generate an image corresponding to a generated area.

[0219] In example (1210), the electronic device (101) can acquire images (1211, 1212, 1213, 1214, 1215) for generating a panoramic image. For example, the images (1211, 1212, 1213, 1214, 1215) can be stitched together. For example, the electronic device (101) can identify parts of the first image and the second image immediately following the first image among the images (1211, 1212, 1213, 1214, 1215) to be stitched together. For example, the electronic device (101) can perform identification of parts of the first image and parts of the second image. At this time, the electronic device (101) may determine a generated region (1219) based on the failure of the identification of the portion of the first image and the portion of the second image. As a non-limiting example, the generated region (1219) may be included in at least a portion (1217) of the image in which all of the images (1211, 1212, 1213, 1214, 1215) are stitched together. For example, at least a portion (1217) may be referred to as the stitched image.

[0220] In example (1220), the electronic device (101) can identify a reference area (1221) that includes a generating area (1219). For example, the reference area (1221) may be an area that extends from the generating area (1219) in a second direction (e.g., horizontal or transverse direction). For example, the electronic device (101) may perform masking on the reference area (1221). For example, the electronic device (101) may create a masked reference area (1223). For example, the masked reference area (1223) may include a non-masking portion (1229) corresponding to the generating area (1219) and a masking portion (1225) different from the generating area (1219).

[0221] In example (1220), an electronic device (101) is shown identifying a reference region (1221) and a masked reference region (1223) from a generating region (1219) to be provided as input to a trained model (250), but the present disclosure is not limited thereto. For example, the electronic device (101) may provide the generating region (1219) to the trained model (250). In other words, the operation of example (1220) may be omitted.

[0222] In example (1230), the electronic device (101) can identify (or extract) scene information (1231) using a masked reference area (1223). For example, the electronic device (101) can identify (or extract) scene information (1231) including at least one word representing at least one object of the non-masked portion (1229) of the masked reference area (1223) by analyzing the masked reference area (1223). For example, the scene information (1231) can be referenced by a subject word.

[0223] In example (1240), the electronic device (101) can generate a prompt (1241) using scene information (1231). For example, the prompt (1241) may include at least one sentence describing the scene of the non-masking portion (1229) using scene information (1231). The scene information (1231) and the prompt (1241) may be referred to as text information. For example, the text information may be an input provided to a trained model (250).

[0224] Specific examples related to example (1230) and example (1240) may be referenced in FIG. 12b.

[0225] FIG. 12b illustrates an example of a method for generating information for stitching images to be used as input to a trained model.

[0226] FIG. 12b illustrates an example of a method for generating text information included in information for stitching images.

[0227] Referring to FIG. 12b, the electronic device (101) can generate an input (1271). For example, the input (1271) may be an example of the masked reference area (1223) of FIG. 12a. However, the present disclosure is not limited thereto. For example, the input (1271) may be an example of the generated area (1219) of FIG. 12a.

[0228] For example, the electronic device (101) may provide an input (1271) to the prompt generation module (1270). For example, the prompt generation module (1270) may be implemented with instructions stored in memory (350). However, the present disclosure is not limited thereto. For example, the prompt generation module (1270) may be implemented with other trained models. In other words, the prompt generation module (1270) may be implemented with an artificial intelligence model. The other trained model may be referred to as the prompt generation model.

[0229] For example, the electronic device (101) can obtain an output (1279) by using a prompt generation module (1270). For example, the output (1279) may be an example of text information. For example, the output (1279) may include scene information (1231) and a prompt (1241) of FIG. 12a. In the example of FIG. 12b, the output (1279) may include “mountain, sky, sunset, lake” as scene information. Also, in the example of FIG. 12b, the output (1279) may include “a mountain is located in the center, and the orange sky with a sunset is reflected in the lake below the mountain” as a prompt.

[0230] Referring again to FIG. 12a, in example (1250), the electronic device (101) may provide information for stitching images to a trained model (250). For example, the information may include a stitched image (1217), a prompt (1241) (and / or, scene information (1231)), and a masked reference area (1223) (or, a generated area (1219)). For example, the trained model (250) may use the stitched image (1217), the prompt (1241) (and / or, scene information (1231)), and the masked reference area (1223) (or, a generated area (1219)) to generate an image (1251) corresponding to the generated area (1219). As a non-limiting example, the stitched image (1217) may be an image in which all of the images (1211, 1212, 1213, 1214, 1215) are stitched, or may be some of the images in which all of the images (1211, 1212, 1213, 1214, 1215) are stitched. A specific example related to this may be referenced in FIG. 12c below.

[0231] FIG. 12c illustrates an example of a method for generating an image corresponding to a generated region using a trained model by providing information for stitching images.

[0232] FIG. 12c illustrates an example of a method for generating an image corresponding to a generation area by providing information to a trained model (250) for stitching images.

[0233] Referring to FIG. 12c, the electronic device (101) may provide the input (1271) of FIG. 12b and the output (1279) of FIG. 12b to the trained model (250). For example, the input (1271) may be an example of the masked reference area (1223) of FIG. 12a. However, the present disclosure is not limited thereto. For example, the input (1271) may be an example of the generated area (1219) of FIG. 12a. Although not illustrated in FIG. 12c, as a non-limiting example, the input (1271) may include the generated area (1219) of FIG. 12a as well as at least a portion (1217) containing the generated area (1219) among the images stitched to date. For example, the output (1279) may include scene information (1231) and prompt (1241) of FIG. 12a.

[0234] For example, the electronic device (101) can generate an image (1281) corresponding to a generation area (1219) using a trained model (250). For example, the image (1281) may be an example of the image (1251) of FIG. 12a. As an example without limitation, the trained model (250) may be a generative artificial intelligence model.

[0235] Referring to FIG. 12c, the trained model (250) can generate an image (1281) that corrects masked parts in the input (1271) or unnatural parts within the input (1271) by using generated area and text information.

[0236] Referring again to FIG. 12a, the electronic device (101) can generate a stitched image (1261) by updating an image (1251) generated using a trained model (250). For example, the stitched image (1261) may be an example of an image where images (1211, 1212, 1213, 1214, 1215) are stitched. For example, a part of the stitched image (1261) may be replaced with an image (1251). As a non-limiting example, the image (1251) of the stitched image (1261) of example (1260) may correspond to at least a part (1217) of the image where images (1211, 1212, 1213, 1214, 1215) are stitched in example (1210). Although not illustrated in FIG. 12a, the electronic device (101) can update the panoramic image and thumbnail image being generated using the stitched image (1261) (or the image (1251) generated using the trained model (250)).

[0237] FIG. 13 illustrates an example of a method for obtaining stitched parts of images using a trained model based on the failure to identify parts of the images to be stitched in order to generate a panoramic image.

[0238] At least some of the above methods of FIG. 13 may be performed by the electronic device (101) of FIG. 3. For example, at least some of the above methods may be controlled by at least one processor (310) of the electronic device (101). In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0239] According to one embodiment, in operation (1310), the electronic device (101) can drive at least one camera (320) to generate a panoramic image. For example, by driving at least one camera (320), the electronic device (101) can sequentially acquire images through at least one camera (320).

[0240] Although not illustrated in FIG. 13, the electronic device (101) can execute a camera application. For example, the electronic device (101) can activate a panoramic shooting mode of the executed camera application. For example, the electronic device (101) can identify an input for generating a panoramic image within the panoramic shooting mode.

[0241] According to one embodiment, in operation (1320), the electronic device (101) can identify motion data of the electronic device (101) through at least one sensor (330). For example, the electronic device (101) can identify motion data while acquiring an image using at least one sensor (330). For example, the motion data can be used to identify the posture (or movement, direction, position) of the electronic device (101) when acquiring an image.

[0242] According to one embodiment, in operation (1330), the electronic device (101) can acquire a first image and a second image following the first image. For example, the electronic device (101) can acquire the first image and the second image while generating the panoramic image.

[0243] According to one embodiment, in operation (1340), the electronic device (101) can compare the second image with the first image. For example, the electronic device (101) can compare the second image with the first image while generating the panoramic image.

[0244] For example, the electronic device (101) can perform the comparison using feature points of the first image and feature points of the second images. For example, the electronic device (101) can identify (or extract) feature points of the first image. For example, the electronic device (101) can identify (or extract) feature points of the second image. For example, feature points of the image can be extracted from the content (or objects) within the image.

[0245] For example, the electronic device (101) can identify, by performing the comparison, the difference value between the feature points of the first image and the feature points of the second image, the number of the feature points of the second image, or whether the content within the second image includes repeated patterns. For example, the electronic device (101) can identify, as a result of the comparison, whether the relative position of the second image with respect to the first image can be determined.

[0246] According to one embodiment, in operation (1350), the electronic device (101) may perform identification of the portion of the first image and the portion of the second image to be stitched to create a panoramic image according to the result of the comparison. While creating the panoramic image, the electronic device (101) may perform identification of the portion of the first image and the portion of the second image to be stitched to create a panoramic image according to the result of the comparison.

[0247] For example, the electronic device (101) can perform the identification of a portion of the first image and a portion of the second image by performing image placement and image stitching according to the result of the comparison. For example, specific details regarding the image placement may be referenced in the operation (415) of FIG. 4 and FIG. 5 to 6b. For example, specific details regarding the image stitching may be referenced in the operation (420) of FIG. 4 and FIG. 7 to 8b.

[0248] According to one embodiment, in operation (1360), the electronic device (101) may obtain information for stitching the first image and the second image based on the failure of the identification. For example, the electronic device (101) may obtain information for stitching the first image and the second image based on the failure of the identification while generating the panoramic image. For example, the electronic device (101) may obtain the information using the motion data, the first image, and the second image identified while acquiring the first image and the second image.

[0249] For example, the electronic device (101) may determine (or estimate) the relative position of an image using motion data, determine that the identification has failed if the difference between parts of the images to be stitched exceeds a threshold value, or if the size of the area of ​​the stitched image located outside the reference angle of view exceeds a reference size. For example, reference may be made to FIGS. 5 to 6b above for an example of the failure of the identification caused by determining the relative position of an image using motion data. For example, reference may be made to FIGS. 7 to 8b above for an example of the failure of the identification caused by the difference between parts of the images to be stitched exceeding a threshold value. For example, reference may be made to FIGS. 9 to 10b above for an example of the failure of the identification caused by the size of the area of ​​the stitched image located outside the reference angle of view exceeding a reference size. For example, if the identification fails, the electronic device (101) can determine the part of the panoramic image where the identification failed as the generation area.

[0250] For example, the information for stitching the first image and the second image may include the generating area. For example, the information for stitching the first image and the second image may include text information describing the generating area. For example, specific details regarding the method of obtaining the information may be referenced in FIG. 11 and FIG. 12a to 12b.

[0251] According to one embodiment, in operation (1370), the electronic device (101) can obtain a portion of the first image and the second image stitched together to be included in the panoramic image by providing the information to the trained model (250). For example, the electronic device (101) can provide the information to the trained model (250) while generating the panoramic image. For example, the electronic device (101) can obtain a portion of the first image and the second image stitched together to be included in the panoramic image while generating the panoramic image.

[0252] For example, the electronic device (101) can generate an image corresponding to a generated area by providing the information to a trained model (250). Generating an image corresponding to a generated area using the trained model (250) may be illustrated in FIG. 11, FIG. 12a, and FIG. 12c described above.

[0253] Although not illustrated in FIG. 13, the electronic device (101) may generate a final panoramic image. For example, the electronic device (101) may generate a final panoramic image and perform processing on the generated final panoramic image. Examples, without limitation, include compression and / or storage of the final panoramic image. In other words, after generating the final panoramic image, the electronic device (101) may compress the final panoramic image to store it in memory (350) and store it in memory (350).

[0254] Although not illustrated in FIG. 13, the electronic device (101) may display a preview image through a display (340). For example, when the electronic device (101) acquires the first image, it may display a preview image representing the first image through the display (340). For example, when the electronic device (101) acquires the second image following the first image, it may display a preview image representing the second image through the display (340). For example, the preview image representing the second image may be updated from the preview image representing the first image. For example, while displaying the preview image, the electronic device (101) may perform at least one of operation (1340), operation (1350), operation (1360), and operation (1370).

[0255] Although not illustrated in FIG. 13, the electronic device (101) may display a thumbnail image through a display (340). For example, after acquiring the first image, the electronic device (101) may display a thumbnail image representing the first image through the display (340). For example, after acquiring the second image following the first image, the electronic device (101) may display a thumbnail image representing an image in which the first image and the second image are stitched together through the display (340). For example, the thumbnail image representing the image in which the first image and the second image are stitched together may be updated from the thumbnail image representing the first image. For example, while displaying the thumbnail image, the electronic device (101) may perform at least one of operation (1340), operation (1350), operation (1360), and operation (1370).

[0256] In FIG. 13, the case in which the second image following the first image is obtained is illustrated, but the present disclosure is not limited thereto. For example, the electronic device (101) may obtain a third image following the second image. For example, the electronic device (101) may perform operations (1340) to (1370) substantially identically with respect to the third image as well.

[0257] FIG. 14 is a block diagram of an electronic device in a network environment according to various embodiments.

[0258] Referring to FIG. 14, in a network environment (1400), an electronic device (1401) may communicate with an electronic device (1402) through a first network (1498) (e.g., a short-range wireless communication network) or with at least one of an electronic device (1404) or a server (1408) through a second network (1499) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (1401) may communicate with the electronic device (1404) through a server (1408). According to one embodiment, the electronic device (1401) may include a processor (1420), memory (1430), input module (1450), sound output module (1455), display module (1460), audio module (1470), sensor module (1476), interface (1477), connection terminal (1478), haptic module (1479), camera module (1480), power management module (1488), battery (1489), communication module (1490), subscriber identification module (1496), or antenna module (1497). In some embodiments, at least one of these components (e.g., connection terminal (1478)) may be omitted from the electronic device (1401), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (1476), camera module (1480), or antenna module (1497)) may be integrated into a single component (e.g., display module (1460)).

[0259] The processor (1420) can, for example, execute software (e.g., program (1440)) to control at least one other component (e.g., hardware or software component) of the electronic device (1401) connected to the processor (1420) and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (1420) can store commands or data received from other components (e.g., sensor module (1476) or communication module (1490)) in volatile memory (1432), process the commands or data stored in volatile memory (1432), and store the resulting data in non-volatile memory (1434). According to one embodiment, the processor (1420) may include a main processor (1421) (e.g., a central processing unit or an application processor) or an auxiliary processor (1423) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (1401) includes a main processor (1421) and an auxiliary processor (1423), the auxiliary processor (1423) may be configured to use less power than the main processor (1421) or to be specialized for a designated function. The auxiliary processor (1423) may be implemented separately from the main processor (1421) or as part thereof.

[0260] The auxiliary processor (1423) may control at least some of the functions or states associated with at least one component of the electronic device (1401) (e.g., display module (1460), sensor module (1476), or communication module (1490)) on behalf of the main processor (1421) while the main processor (1421) is in an inactive (e.g., sleep) state, or together with the main processor (1421) while the main processor (1421) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (1423) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (1480) or communication module (1490)). According to one embodiment, the auxiliary processor (1423) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (1401) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (1408)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0261] The memory (1430) can store various data used by at least one component of the electronic device (1401) (e.g., processor (1420) or sensor module (1476)). The data may include, for example, input data or output data for software (e.g., program (1440)) and related commands. The memory (1430) may include volatile memory (1432) or non-volatile memory (1434).

[0262] The program (1440) may be stored as software in memory (1430) and may include, for example, an operating system (1442), middleware (1444), or an application (1446).

[0263] The input module (1450) can receive commands or data to be used for a component of the electronic device (1401) (e.g., processor (1420)) from outside the electronic device (1401) (e.g., user). The input module (1450) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0264] The sound output module (1455) can output a sound signal to the outside of the electronic device (1401). The sound output module (1455) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0265] The display module (1460) can visually provide information to an external (e.g., user) of the electronic device (1401). The display module (1460) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (1460) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0266] The audio module (1470) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (1470) can acquire sound through an input module (1450) or output sound through an audio output module (1455) or an external electronic device (e.g., electronic device (1402)) (e.g., speaker or headphones) that is directly or wirelessly connected to the electronic device (1401).

[0267] The sensor module (1476) can detect the operating state of the electronic device (1401) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (1476) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0268] The interface (1477) may support one or more specified protocols that can be used for the electronic device (1401) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (1402)). According to one embodiment, the interface (1477) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0269] The connection terminal (1478) may include a connector through which the electronic device (1401) can be physically connected to an external electronic device (e.g., electronic device (1402)). According to one embodiment, the connection terminal (1478) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0270] The haptic module (1479) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that can be perceived by the user through tactile or kinesthetic senses. According to one embodiment, the haptic module (1479) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0271] The camera module (1480) can capture still images and video. According to one embodiment, the camera module (1480) may include one or more lenses, image sensors, image signal processors, or flashes.

[0272] The power management module (1488) can manage power supplied to the electronic device (1401). According to one embodiment, the power management module (1488) may be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0273] The battery (1489) can supply power to at least one component of the electronic device (1401). According to one embodiment, the battery (1489) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0274] The communication module (1490) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (1401) and an external electronic device (e.g., electronic device (1402), electronic device (1404), or server (1408)), and the performance of communication through the established communication channel. The communication module (1490) may include one or more communication processors that operate independently of the processor (1420) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (1490) may include a wireless communication module (1492) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (1494) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (1404) through a first network (1498) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (1499) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (1492) can identify or authenticate the electronic device (1401) within a communication network such as the first network (1498) or the second network (1499) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (1496).

[0275] The wireless communication module (1492) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (1492) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (1492) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (1492) can support various requirements specified in the electronic device (1401), external electronic device (e.g., electronic device (1404)), or network system (e.g., second network (1499)). According to one embodiment, the wireless communication module (1492) may support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0276] An antenna module (1497) can transmit a signal or power to an external source (e.g., an external electronic device) or receive it from an external source. According to one embodiment, the antenna module (1497) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (1497) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (1498) or a second network (1499), may be selected from the plurality of antennas, for example, by a communication module (1490). A signal or power may be transmitted or received between the communication module (1490) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (1497).

[0277] According to various embodiments, the antenna module (1497) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0278] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0279] According to one embodiment, commands or data may be transmitted or received between an electronic device (1401) and an external electronic device (1404) through a server (1408) connected to a second network (1499). Each of the external electronic devices (1402, or 1404) may be the same or a different type of device as the electronic device (1401). According to one embodiment, all or part of the operations performed on the electronic device (1401) may be performed on one or more of the external electronic devices (1402, 1404, or 1408). For example, if the electronic device (1401) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (1401) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (1401). The electronic device (1401) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (1401) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (1404) may include an Internet of Things (IoT) device. The server (1408) may be an intelligent server using machine learning and / or neural networks.According to one embodiment, an external electronic device (1404) or server (1408) may be included within the second network (1499). The electronic device (1401) may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0280] As described above, the electronic device (101) may include at least one sensor (330). The electronic device (101) may include at least one camera (320). The electronic device (101) may include a memory (350) that stores instructions and includes one or more storage media. The electronic device (101) may include at least one processor (310) that includes a processing circuit. The instructions may cause the electronic device (101) to drive the at least one camera (320) to generate a panoramic image when the at least one processor (310) is executed individually or collectively. The instructions may cause the electronic device (101) to identify motion data of the electronic device (101) through the at least one sensor (330) when the at least one processor (310) is executed individually or collectively. The above instructions may cause the electronic device (101) to acquire a first image and a second image following the first image while generating the panoramic image, when the at least one processor (310) is executed individually or collectively. The above instructions may cause the electronic device (101) to compare the second image with the first image while generating the panoramic image, when the at least one processor (310) is executed individually or collectively. The above instructions may cause the electronic device (101) to perform identification of the portion of the first image to be stitched to generate the panoramic image and the portion of the second image to be stitched to generate the panoramic image, based on the result of the comparison, while generating the panoramic image, when the at least one processor (310) is executed individually or collectively.The above instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to obtain information for stitching the first image and the second image using the motion data, the first image, and the second image identified during the acquisition of the first image and the second image based on the failure of identification while generating the panoramic image. The above instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to obtain the portion of the first image and the second image stitched to be included in the panoramic image by providing the information to a trained model while generating the panoramic image.

[0281] According to one embodiment, the instructions may cause the electronic device (101) to determine whether the relative position of the second image with respect to the first image can be determined when the at least one processor (310) is executed individually or collectively. The instructions may cause the electronic device (101) to determine the relative position of the second image using the motion data identified during the acquisition of the first image and the second image, based on the finding that the result of the comparison cannot determine the relative position of the second image. The above instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to determine the identification of the portion of the first image and the portion of the second image as a failure, based on determining the relative position of the second image using the motion data identified while acquiring the first image and the second image, based on identifying that the result of the comparison cannot determine the relative position of the second image.

[0282] According to one embodiment, when the instructions are executed individually or collectively by at least one processor (310), the electronic device (101) may be caused to identify the result of the comparison as being unable to determine the relative position of the second image, depending on the difference value between the feature points of the first image and the feature points of the second image that exceeds a reference value, the number of the feature points of the second image that is less than a reference number, or the content within the second image that includes repeated patterns.

[0283] According to one embodiment, the instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to store in the memory (350) a generation region determined using the motion data, the first image, and the second image identified during the acquisition of the first image and the second image, based on identifying that the result of the comparison cannot determine the relative position of the second image. The instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to acquire a third image following the second image. The instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to compare the third image with the second image. The above instructions may cause the electronic device (101), when executed individually or collectively by at least one processor (310), to obtain the information including the generated region stored in the memory (350) based on identifying that the result of the comparison between the third image and the second image can determine the relative position of the third image with respect to the second image. The generated region may correspond to the portion where the first image and the second image of the panoramic image are stitched together, which will be generated using the trained model.

[0284] According to one embodiment, the electronic device (101) may include a display (340). The instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to perform the following: comparing the second image with the first image, performing the identification of the portion of the first image and the portion of the second image, and obtaining the information for stitching the first image and the second image, while displaying through the display (340) a second thumbnail image representing an image in which the first image and the second image are stitched according to the relative position of the second image determined using the motion data identified while acquiring the first image and the second image. Within the panoramic image, the stitched image may be replaced with the portion in which the first image and the second image are stitched, which is acquired from the trained model.

[0285] According to one embodiment, the instructions may cause the electronic device (101), when executed individually or collectively by the at least one processor (310), to perform the identification of the portion of the first image and the portion of the second image using the relative position of the second image, based on identifying that the result of the comparison can determine the relative position of the second image. The instructions may cause the electronic device (101), when executed individually or collectively by the at least one processor (310), to obtain the panoramic image including the portion where the first image and the second image are stitched together, by bypassing the provision of the information to the trained model based on the success of the identification.

[0286] According to one embodiment, the instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to identify whether the difference between the periphery of the portion of the first image and the periphery of the portion of the second image exceeds a threshold value, depending on the relative position of the second image with respect to the first image. The instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to determine the identification of the portion of the first image and the portion of the second image as a failure based on identifying that the difference exceeds the threshold value.

[0287] According to one embodiment, the instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to identify an area where the second image overlaps the first image according to the relative position of the second image with respect to the first image. The instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to identify the difference within the area using the first color data of the first image and the second color data of the second image.

[0288] According to one embodiment, the edge of the portion of the first image and the edge of the portion of the second image may be defined by a boundary within the region where the sum of the difference values ​​of the first color data and the second color data has a minimum value.

[0289] According to one embodiment, the instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to store in the memory (350) a generated region including the boundary between the edge of the portion of the first image and the edge of the portion of the second image, based on identifying that the difference exceeds the threshold value. The instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to acquire a third image following the second image. The instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to acquire the information including the generated region including the boundary stored in the memory (350), based on that the third image does not include at least a portion of the boundary. The above-mentioned generated area may correspond to the portion where the first image and the second image of the panorama image are stitched together, which will be generated using the above-mentioned trained model.

[0290] According to one embodiment, the electronic device (101) may include a display (340). The instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to perform the following: comparing the second image with the first image, performing the identification of the portion of the first image and the portion of the second image, and obtaining the information for stitching the first image and the second image, while displaying through the display (340) a first thumbnail image representing the first image and a second thumbnail image representing the first image and the second image stitched according to the relative position of the second image. Within the panoramic image, the stitched image may be replaced with the portion of the first image and the second image stitched obtained from the trained model.

[0291] According to one embodiment, the instructions may cause the electronic device (101), when executed individually or collectively by the at least one processor (310), to perform the identification of the portion of the first image and the portion of the second image using the edge of the portion of the first image and the edge of the portion of the second image, based on identifying that the difference is below the threshold value. The instructions may cause the at least one processor (310), when executed individually or collectively by the success of the identification, to obtain the panoramic image including the portion where the first image and the second image are stitched together by bypassing the provision of the information to the trained model.

[0292] According to one embodiment, the instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to identify a first reference angle of view in a first direction defined by the first image as the first image is acquired. The instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to generate an image in which the first image and the second image are stitched together according to the relative position of the second image with respect to the first image. The instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to identify a second reference angle of view in a second direction perpendicular to the first direction defined by the stitched image. The above instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to identify whether the size of the area where the stitched image is not located within the reference angle area defined by the first reference angle and the second reference angle exceeds the reference size. The above instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to determine the identification of the portion of the first image and the portion of the second image as a failure based on identifying that the size exceeds the reference size.

[0293] According to one embodiment, the instructions may cause the electronic device (101), when executed individually or collectively by the at least one processor (310), to obtain information including a generated area including the area where the stitched image is not located, based on identifying that the size exceeds the reference size. The generated area may correspond to the portion where the first image and the second image of the panoramic image are stitched together, which will be generated using the trained model.

[0294] According to one embodiment, the instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to determine the identification of the portion of the first image and the portion of the second image as successful based on identifying that the size is less than or equal to the reference size while the thumbnail image representing the first image is displayed through the display (340). The instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to obtain the panoramic image including the portion where the first image and the second image are stitched together by bypassing the provision of the information to the trained model based on the success of the identification while the thumbnail image representing the first image is displayed through the display (340).

[0295] According to one embodiment, the information for stitching the first image and the second image may include a generated area included in the portion where the first image and the second image of the panoramic image to be generated from the trained model are stitched, and text information describing the generated area.

[0296] According to one embodiment, the text information may be generated using another trained model for generating sentences describing an image.

[0297] According to one embodiment, the electronic device (101) may include a display (340). The instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to display through the display (340) a first thumbnail image representing an image including another portion where the first image and the second image are stitched together while generating the panoramic image. The instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to provide the information to the trained model to obtain the portion where the first image and the second image are stitched together, and then to display through the display (340) a second thumbnail image representing another image in which the other portion where the first image and the second image are stitched together is replaced by the portion where the first image and the second image are stitched together.

[0298] According to one embodiment, the instructions may cause the electronic device (101) to execute a camera application when the at least one processor (310) is executed individually or collectively. The instructions may cause the electronic device (101) to drive the at least one camera (320) within a panoramic shooting mode of the executed camera application when the at least one processor (310) is executed individually or collectively.

[0299] As described above, the electronic device (101) may include at least one sensor (330). The electronic device (101) may include at least one camera (320). The electronic device (101) may include a display (340). The electronic device (101) may include a memory (350) that stores instructions and includes one or more storage media. The electronic device (101) may include at least one processor (310) that includes a processing circuit. The instructions may cause the electronic device (101) to display a first image among a plurality of images through the display (340) while acquiring a plurality of images through the at least one camera (320) based on a panoramic shooting mode, when the at least one processor (310) is executed individually or collectively. The above instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to determine whether the relative position of the second image with respect to the first image can be determined based on the first image and the second image following the first image among the plurality of images while the first image is displayed through the display (340). The above instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to determine the relative position of the second image with respect to the first image based on motion data of the electronic device (101) obtained through the at least one sensor (330), depending on the failure to determine the relative position of the second image with respect to the first image.The above instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to display, through the display (340), a first stitched image in which the second image is at least partially superimposed on the first image using the relative position of the second image. The above instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to identify whether the relative position of the third image with respect to the second image can be determined based on the second image and the third image following the second image among the plurality of images while the first stitched image is displayed through the display (340). The above instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to provide at least a portion of the first stitched image to a trained model stored in the memory (350) upon identifying success in determining the relative position of the third image with respect to the second image. The above instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to use the trained model to generate an image representing an area in which the second image is at least partially superimposed on the first image. The above instructions may cause the electronic device (101), when the at least one processor (310) is executed individually or collectively, to display through the display (340) a second stitched image in which the third image is at least partially superimposed on the first stitched image, in which at least a portion has been replaced by the generated image.

[0300] Referring to the foregoing, in performing real-time image stitching to generate a panoramic image, the present disclosure can eliminate distortion that may occur in the acquired images and in the parts for stitching the acquired images without user input. For example, the present disclosure can correct a panoramic image using an image generated using a trained model. In the corrected panoramic image (or final panoramic image), unnatural parts (e.g., color differences, broken lines) can be suppressed (or eliminated), and loss of field of view can be suppressed.

[0301] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs from the description below.

[0302] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0303] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B, or C” may each include any one of the items listed together in the corresponding phrase, or any combination thereof. Terms such as “first,” “second,” or “first” or “second” may be used simply to distinguish a component from another corresponding component and do not limit the components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0304] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0305] Various embodiments of the present document may be implemented as software (e.g., program (1440)) comprising one or more instructions stored in a storage medium (e.g., internal memory (1436) or external memory (1438)) readable by a machine (e.g., electronic device (1401)). For example, a processor (e.g., processor (1420)) of the machine (e.g., electronic device (1401)) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0306] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0307] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device, At least one sensor; At least one camera; Memory that stores instructions and includes one or more storage media; and It includes at least one processor comprising a processing circuit, and When the above instructions are executed individually or collectively by at least one processor, the electronic device: To generate a panoramic image, drive the at least one camera; Identifying motion data of the electronic device through the at least one sensor; and While generating the above panorama image: A first image and a second image following the first image are obtained, and Compare the second image above with the first image, and Based on the result of the above comparison, identification is performed of the portion of the first image to be stitched to generate the panoramic image and the portion of the second image to be stitched to generate the panoramic image, and Based on the failure of the above identification, information for stitching the first image and the second image is obtained using the motion data identified while acquiring the first image and the second image, the first image, and the second image, and By providing the above information to a trained model, causing to obtain a stitched portion of the first image and the second image to be included in the panoramic image. Electronic device.

2. In Claim 1, When the above instructions are executed individually or collectively by at least one processor, the electronic device: Identifying whether the relative position of the second image with respect to the first image can be determined; Based on identifying that the result of the above comparison cannot determine the relative position of the second image, Using the motion data identified while acquiring the first image and the second image, the relative position of the second image is determined, and Based on determining the relative position of the second image using the motion data identified while acquiring the first image and the second image, causing the identification of the part of the first image and the part of the second image to be determined as a failure. Electronic device.

3. In Claim 2, When the above instructions are executed individually or collectively by at least one processor, the electronic device: A difference value between the feature points of the first image and the feature points of the second image that exceeds a reference value, the number of the feature points of the second image that is less than a reference number, or, depending on the content within the second image including repeated patterns, causing the result of the comparison to be identified as being unable to determine the relative position of the second image. Electronic device.

4. In Claim 2, When the above instructions are executed individually or collectively by at least one processor, the electronic device: Based on identifying that the result of the above comparison cannot determine the relative position of the second image, a generation area determined using the motion data identified during the acquisition of the first image and the second image, the first image, and the second image is stored in the memory, and Obtain a third image following the second image above, and Compare the above third image with the above second image, and Based on identifying that the result of the comparison between the third image and the second image can determine the relative position of the third image with respect to the second image, the result causes to obtain the information including the generated region stored in the memory, and The above-mentioned generated region corresponds to the portion where the first image and the second image of the panorama image are stitched together, to be generated using the above-mentioned trained model. Electronic device.

5. In Claim 4, The above electronic device includes a display, When the above instructions are executed individually or collectively by at least one processor, the electronic device: After displaying a first thumbnail image representing the first image, while displaying through the display a second thumbnail image representing an image in which the first image and the second image are stitched together according to the relative position of the second image determined using the motion data identified while acquiring the first image and the second image: Causing to perform the following: comparing the second image with the first image; performing the identification of the portion of the first image and the portion of the second image; and obtaining the information for stitching the first image and the second image. In the above panoramic image, the stitched image is replaced by the portion where the first image and the second image obtained from the trained model are stitched together. Electronic device.

6. In Claim 2, When the above instructions are executed individually or collectively by at least one processor, the electronic device: Based on identifying that the result of the above comparison can determine the relative position of the second image: Using the relative position of the second image, the identification of the portion of the first image and the portion of the second image is performed, and Based on the success of the above identification, by bypassing the provision of the above information to the trained model, causing the panoramic image including the portion where the first image and the second image are stitched together to be obtained, Electronic device.

7. In Claim 1, When the above instructions are executed individually or collectively by at least one processor, the electronic device: Identifying whether the difference between the periphery of the portion of the first image and the periphery of the portion of the second image exceeds a threshold value based on the relative position of the second image with respect to the first image; and Based on identifying that the above difference exceeds the above threshold value, causing the identification of the above part of the first image and the above part of the second image to be determined as the above failure, Electronic device.

8. In Claim 7, When the above instructions are executed individually or collectively by at least one processor, the electronic device: Identifying an area where the second image overlaps the first image according to the relative position of the second image with respect to the first image; and Within the above area, using the first color data of the first image and the second color data of the second image, causing to identify the difference, Electronic device.

9. In Claim 8, The edge of the portion of the first image and the edge of the portion of the second image are defined by a boundary within the region where the sum of the difference values ​​of the first color data and the second color data has a minimum value. Electronic device.

10. In Claim 9, When the above instructions are executed individually or collectively by at least one processor, the electronic device: Based on identifying that the above difference exceeds the above threshold value, a generated region including the boundary between the above edge of the above part of the first image and the above edge of the above part of the second image is stored in the memory, and Obtain a third image following the second image above, and Based on the fact that the third image does not include at least a portion of the boundary, causing to obtain the information including the generated region including the boundary stored in the memory, and The above-mentioned generated region corresponds to the portion where the first image and the second image of the panorama image are stitched together, to be generated using the above-mentioned trained model. Electronic device.

11. In Claim 10, The above electronic device includes a display, When the above instructions are executed individually or collectively by at least one processor, the electronic device: After displaying a first thumbnail image representing the first image, while displaying through the display a second thumbnail image representing an image in which the first image and the second image are stitched together according to the relative position of the second image: Causing to perform the following: comparing the second image with the first image; performing the identification of the portion of the first image and the portion of the second image; and obtaining the information for stitching the first image and the second image. In the above panoramic image, the stitched image is replaced by the portion where the first image and the second image obtained from the trained model are stitched together. Electronic device.

12. In Claim 7, When the above instructions are executed individually or collectively by at least one processor, the electronic device: Based on identifying that the above difference is below the above threshold value: Using the edge of the portion of the first image and the edge of the portion of the second image, the identification of the portion of the first image and the portion of the second image is performed, and Based on the success of the above identification, by bypassing the provision of the above information to the trained model, causing the panoramic image including the portion where the first image and the second image are stitched together to be obtained, Electronic device.

13. In Claim 1, When the above instructions are executed individually or collectively by at least one processor, the electronic device: Upon acquiring the first image, a first reference angle of view in a first direction defined by the first image is identified; Based on the relative position of the second image with respect to the first image, an image is generated in which the first image and the second image are stitched together; Identifying a second reference angle of view in a second direction perpendicular to the first direction defined by the stitched image above; Identifying whether the size of the area where the stitched image is not located within the reference angle area defined by the first reference angle and the second reference angle exceeds the reference size; and Based on identifying that the above size exceeds the above standard size, causing the identification of the above part of the first image and the above part of the second image to be determined as the above failure, Electronic device.

14. In Claim 13, When the above instructions are executed individually or collectively by at least one processor, the electronic device: Based on identifying that the above size exceeds the above reference size, causing to obtain the above information including a generating area including the area where the stitched image is not located, and The above-mentioned generated region corresponds to the portion where the first image and the second image of the panorama image are stitched together, to be generated using the above-mentioned trained model. Electronic device.

15. In Claim 13, When the above instructions are executed individually or collectively by at least one processor, the electronic device: While displaying a thumbnail image representing the first image through the display: Based on identifying that the above size is less than or equal to the above reference size, determining the identification of the above portion of the first image and the above portion of the second image as successful, and Based on the success of the above identification, by bypassing the provision of the above information to the trained model, causing the panoramic image including the portion where the first image and the second image are stitched together to be obtained, Electronic device.

Images