Electronic device and method for operating electronic device

Abstract

This electronic device comprises a first camera having a first angle of view, a second camera having a second angle of view, a memory, and a processor. The electronic device may: obtain a main object area including a main object and the surrounding area from a first preview image obtained through the first camera; detect, from a second preview image obtained through the second camera, a sub-object included in an area different from the main object area and the surrounding area and moving over a plurality of frames; obtain light source information about the position of a light source in a space including the electronic device on the basis of the position of the sub-object, the height of the sub-object, and the length of the shadow cast thereby; and provide time information about a frame in which the shadow of the sub-object, obtained on the basis of the light source information, the position of the sub-object, and the height of the sub-object, overlaps the main object area.

Description

Electronic device and method of operation of electronic device

[0001] The present disclosure relates to an electronic device and a method of operating the electronic device. Specifically, it relates to an electronic device and a method of operating the electronic device for preventing an object to be photographed from being obscured by an external shadow.

[0002] With the recent advancement of camera technology, cameras are being incorporated into various electronic devices, and technologies for acquiring video images through them are being widely used.

[0003] In addition, with the advancement of technology, electronic devices include multiple cameras with various viewing angles, and images captured at different viewing angles can be obtained through each camera.

[0004] However, a problem may arise where the quality of the image the user intends to capture is lowered due to unwanted objects or shadows of such objects caused by the sun or lighting.

[0005] One embodiment of the present disclosure provides an electronic device. The electronic device may include a first camera having a first field of view (FOV). The electronic device may include a second camera having a second field of view wider than the first field of view. The electronic device may include a memory in which a program or at least one instruction is stored. The electronic device may include at least one processor. By having at least one processor execute the program or at least one instruction stored in the memory individually or collectively, the electronic device may obtain a main object area containing a main object and a surrounding area adjacent to the main object area from a first preview image obtained through the first camera. The electronic device may detect a sub-object that is included in an area different from the main object area and the surrounding area and moves across a plurality of frames from a second preview image obtained through the second camera. The electronic device may obtain light source information regarding the location of a light source in a space containing the electronic device based on the position of the sub-object, the height of the sub-object, and the length of the shadow cast by the sub-object. The electronic device can provide time information for the frame in which the shadow of a sub-object overlaps with the main object area, obtained based on light source information, the position of the sub-object, and the height of the sub-object.

[0006] In one embodiment of the present disclosure, a method of operating an electronic device may be provided. The method of operating an electronic device may include the step of acquiring a main object area containing a main object and a surrounding area adjacent to the main object area from a first preview image acquired through a first camera having a first field of view (FOV). The method of operating an electronic device may include the step of detecting a sub-object that is included in an area different from the main object area and the surrounding area and moves across a plurality of frames from a second preview image acquired through a second camera having a second field of view wider than the first field of view. The method of operating an electronic device may include the step of acquiring light source information regarding the location of a light source in a space containing the electronic device based on the position of the sub-object, the height of the sub-object, and the length of the shadow cast by the sub-object. The method of operating an electronic device may include the step of providing time information regarding a frame in which the shadow of the sub-object overlaps with the main object area, acquired based on the light source information, the position of the sub-object, and the height of the sub-object.

[0007] In one embodiment of the present disclosure, a computer-readable recording medium may be provided on which a program for performing at least one of the embodiments of the method of operating the disclosed electronic device is recorded on a computer.

[0008] The technical problems to be solved in this document are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure belongs from the description below.

[0009] The present disclosure may be understood from the combination of the following detailed description and the accompanying drawings, where reference numerals denote structural elements.

[0010] FIG. 1 is a drawing for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0011] FIG. 2 is a block diagram for explaining the configuration of an electronic device according to one embodiment of the present disclosure.

[0012] FIG. 3 is a flowchart for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0013] FIG. 4 is a flowchart illustrating the operation of obtaining two-dimensional shadow information to obtain the length of a shadow by a sub-object according to one embodiment of the present disclosure.

[0014] FIG. 5 is a diagram illustrating the operation of obtaining two-dimensional shadow information to obtain the length of a shadow by a sub-object according to one embodiment of the present disclosure.

[0015] FIG. 6 is a flowchart illustrating an operation to acquire light source information based on three-dimensional position information of a sub-object, the height of the sub-object, and the length of the shadow according to one embodiment of the present disclosure.

[0016] FIG. 7 is a diagram illustrating an operation to obtain three-dimensional first sub-position information of a sub-object using depth according to one embodiment of the present disclosure.

[0017] FIG. 8 is a diagram illustrating the operation of obtaining three-dimensional second sub-position information of a sub-object using a top view obtained through inverse mapping according to one embodiment of the present disclosure.

[0018] FIG. 9 is a diagram illustrating an operation to obtain the length of a shadow by a sub-object based on two-dimensional shadow information and three-dimensional position information according to one embodiment of the present disclosure.

[0019] FIG. 10 is a diagram illustrating an operation to acquire light source information based on three-dimensional position information of a sub-object, the height of the sub-object, and the length of the shadow according to one embodiment of the present disclosure.

[0020] FIG. 11 is a diagram illustrating an operation to obtain an overlap prediction time when a sub-object in the current frame does not overlap with a main object area, according to one embodiment of the present disclosure.

[0021] FIG. 12 is a diagram illustrating an operation for obtaining an overlap prediction time according to one embodiment of the present disclosure.

[0022] FIG. 13 is a diagram illustrating an operation to perform time correction or frame correction according to whether the second overlap end time is shorter than the reference time when a sub-object overlaps with the main object area in the current frame, according to one embodiment of the present disclosure.

[0023] FIG. 14 is a diagram illustrating an operation to perform time correction when a second overlap end time is shorter than a reference time, in the case where a sub-object overlaps with a main object area in the current frame according to one embodiment of the present disclosure.

[0024] FIG. 15 is a diagram illustrating an operation to perform frame correction according to one embodiment of the present disclosure, in which, when a sub-object overlaps with a main object area in the current frame, the second overlap end time is equal to or longer than the reference time.

[0025] FIG. 16 is a drawing for explaining the operation when a user uses a head-mounted display device according to one embodiment of the present disclosure.

[0026] The terms used in this disclosure will be briefly explained, and an embodiment of this disclosure will be described in detail.

[0027] Throughout this disclosure, unless specifically stated otherwise, "or" is inclusive and not exclusive. Accordingly, "A or B" may mean "A, B, or both" unless clearly indicated otherwise by the context.

[0028] In the present disclosure, the expression “at least one of a, b, or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “a, b, and c all”, or variations thereof.

[0029] The terms used in this disclosure have been selected to be as widely used as possible, taking into account the functions in the embodiments of this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description section of the relevant embodiments of this disclosure. Therefore, the terms used in this disclosure should be defined not merely by their names, but based on their meanings and the content throughout this disclosure.

[0030] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by those skilled in the art as described in this specification.

[0031] Throughout this disclosure, when a part is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "...part," "module," etc., as used in this disclosure refer to a unit that processes at least one function or operation, and may be implemented in hardware or software, or as a combination of hardware and software.

[0032] The expression “configured to” as used in this disclosure may be replaced, depending on the context, with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” may not necessarily mean only “specifically designed to” in hardware. Instead, in some situations, the expression “system configured to” may mean that the system is “capable of” together with other devices or components. For example, the phrase “a processor configured (or set) to perform A, B, and C” may mean a dedicated processor for performing said operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or an application processor) capable of performing said operations by executing one or more software programs stored in memory.

[0033] In addition, when a component is described in the present disclosure as being “connected” or “connected” to another component, it should be understood that the component may be directly connected to or directly connected to the other component, but unless otherwise specifically stated, it may also be connected or connected through another component in between.

[0034] It should be understood that the blocks in each flowchart and combinations of flowcharts can be executed by one or more computer programs containing computer-executable instructions. One or more computer programs may be stored all in a single memory or may be partitioned and stored in multiple different memories.

[0035] All functions or operations described in this document may be processed by a single processor or a combination of multiple processors.

[0036] Functions related to artificial intelligence according to the present disclosure are operated through processors and memory. One or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if one or more processors are dedicated artificial intelligence processors, the dedicated artificial intelligence processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0037] The predefined rules of operation or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that a predefined rules of operation or artificial intelligence models configured to perform desired characteristics (or objectives) are created by a basic artificial intelligence model being trained using a number of training data by a learning algorithm. Such learning may be performed on the electronic device itself in which the artificial intelligence model according to the present disclosure is used, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0038] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through operations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. The artificial neural network may include a Deep Neural Network (DNN), such as a Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), or Deep Q-Networks, but is not limited to the examples mentioned above.

[0039] Embodiments of the present disclosure are described below with reference to the attached drawings so that those skilled in the art can easily implement them. However, an embodiment of the present disclosure may be implemented in various different forms and is not limited to the embodiment described herein. Furthermore, in order to clearly explain an embodiment of the present disclosure in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the present disclosure are denoted by similar reference numerals.

[0040] Embodiments of the present disclosure will be described in detail below with reference to the drawings.

[0041] FIG. 1 is a drawing for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0042] Referring to FIG. 1, in one embodiment of the present disclosure, FIG. 1 illustrates an electronic device (100), a first camera (110) and a second camera (120) included in the electronic device (100). FIG. 1 also illustrates a first preview image (200) obtained through the first camera (110) and a second preview image (300) obtained through the second camera (120).

[0043] In one embodiment of the present disclosure, the electronic device (100) may be implemented as an electronic device of various shapes, such as a mobile device, a smartphone, a laptop computer, a tablet PC, a wearable device, and a head-mounted display (HMD) device.

[0044] In one embodiment of the present disclosure, the electronic device (100) may include a first camera (110) having a first field of view (FOV, 111) and a second camera (120) having a second field of view (121) wider than the first field of view (111). Herein, "field of view" may mean an angular range of a scene that the camera can acquire as an image through a lens.

[0045] In one embodiment of the present disclosure, the first camera (110) may be a camera set as default when a user using the electronic device (100) executes a shooting mode to capture a specific scene through the electronic device (100).

[0046] In one embodiment of the present disclosure, since the second angle of view (121) of the second camera (120) is wider than the first angle of view (111) of the first camera (110), the second preview image (300) obtained through the second camera (120) may include a scene with a wider angle than the first preview image (200) obtained through the first camera (110).

[0047] In this case, "preview image" may refer to an image provided in real time to show a scene being captured through a camera in advance via a display, etc., before capturing a scene through a camera and acquiring the captured image.

[0048] In one embodiment of the present disclosure, the first preview image (200) may include a main object (210). The main object (210) may refer to an object that the user of the electronic device (100) intends to photograph through the first camera (110).

[0049] In one embodiment of the present disclosure, the main object (210) may mean an object located in the central part of the first preview image (200), an object having a size greater than or equal to a preset size among the objects detected in the first preview image (200), or an object that corresponds to one of the preset reference objects in the first preview image (200).

[0050] In one embodiment of the present disclosure, the first preview image (200) may include a main object area (220) including a main object (210) and a surrounding area (230) adjacent to the main object area (220).

[0051] In one embodiment of the present disclosure, the user can confirm through the first preview image (200) that there are no obstacles obscuring the main object area (220) and obtain a first captured image by taking the first preview image (200) through the first camera (110).

[0052] However, a first captured image may be obtained in which the main object area (220) is obscured by an object not included in the first preview image (200). Hereinafter, for convenience of explanation, an object not included in the first preview image (200) may be referred to as a sub-object (320).

[0053] Specifically, when an electronic device (100) obtains a first captured image by capturing a scene by referring to a first preview image (200), the main object area (220) may be obscured by the shadow of a sub-object (320) that is not located in the main object area (220) and the surrounding area (230) within the first preview image (200). Accordingly, the electronic device (100) may obtain a first captured image with degraded quality, in which the shadow (330) caused by the sub-object (320) not included in the main object area (220) and the surrounding area (230) overlaps with the main object area (220).

[0054] The electronic device (100) of the present disclosure can provide the user with information about the time at which the shadow (330) of a sub-object (320) is predicted to overlap with the main object area (220) due to the movement of the sub-object (320).

[0055] In one embodiment of the present disclosure, the electronic device (100) can detect a sub-object (320) based on a second preview image (300) that includes a scene with a wider angle than the first preview image (200). The sub-object (320) may be an object located in an area different from the main object area (220) and the surrounding area (230) within the second preview image (300).

[0056] In one embodiment of the present disclosure, the sub-object (320) may be an object that moves across a plurality of frames. As the position of the sub-object (320) changes across a plurality of frames, the position and size of the shadow (330) caused by the sub-object (320) by the light source of the space containing the electronic device (100) may also change.

[0057] In one embodiment of the present disclosure, the electronic device (100) can obtain light source information regarding the position of a light source in a space containing the electronic device (100) by considering the position of a sub-object (320), the length of a shadow (330) caused by the sub-object (320), the height of the sub-object (320), etc.

[0058] In one embodiment of the present disclosure, the electronic device (100) can obtain the length and position of the shadow (330) caused by the sub-object (320) according to the obtained light source information, the position of the sub-object (320), the height of the sub-object (320), etc.

[0059] In one embodiment of the present disclosure, the electronic device (100) can calculate the time at which the shadow (330) of the sub-object (320) is predicted to overlap with the main object area (220) due to the movement of the sub-object (320). Additionally, the electronic device (100) can calculate the time at which the shadow (330) of the sub-object (320) that overlaps with the main object area (220) due to the movement of the sub-object (320) is predicted not to overlap with the main object area (220).

[0060] In one embodiment of the present disclosure, the electronic device (100) may provide the user with information regarding the time at which the shadow (330) of a sub-object (320) is predicted to overlap with the main object area (220), or information regarding the time at which the shadow (330) of a sub-object (320) that overlaps with the main object area (220) is predicted not to overlap with the main object area (220), in the form of an alarm (400) (e.g., "The object to be photographed may be obscured within 5 seconds."). At this time, the alarm (400) may be provided as a voice or displayed on a display included in the electronic device (100).

[0061] Additionally, the electronic device (100) may determine the time to acquire a first captured image by taking a first preview image (200) through the first camera (110), taking into account the time when the shadow (330) of the sub-object (320) is predicted to overlap with the main object area (220) or the time when the shadow (330) of the sub-object (320) that overlaps with the main object area (220) is predicted not to overlap with the main object area (220).

[0062] Through this, the electronic device (100) can prevent the user from being provided with a first captured image of degraded quality in which the shadow (330) of the sub-object (320) overlaps with the main object area (220).

[0063] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood from the present disclosure by those skilled in the art to which the present disclosure pertains.

[0064] Hereinafter, the electronic device (100) and the method of operation of the electronic device (100) according to the present disclosure will be described later in FIGS. 2 to FIGS. 16 below.

[0065] FIG. 2 is a block diagram for explaining the configuration of an electronic device according to one embodiment of the present disclosure.

[0066] Referring to FIGS. 1 and 2, in one embodiment of the present disclosure, an electronic device (100) may include a display (130), a first camera (110), a second camera (120), a memory (140), at least one processor (150), an inertial measurement unit (IMU, 160), a Global Positioning System (GPS) device (170), an input / output interface (180), and a communication interface (190).

[0067] However, not all of the components shown in FIG. 2 are essential components. The electronic device (100) may be implemented with more components than those shown in FIG. 2, or with fewer components.

[0068] A display (130), a first camera (110), a second camera (120), a memory (140), at least one processor (150), an inertial measurement unit (160), a GPS device (170), an input / output interface (180), and a communication interface (190) included in the electronic device (100) can each be electrically connected to one another.

[0069] In one embodiment of the present disclosure, at least one processor (150) can control a display (130) to display an image or video.

[0070] In one embodiment of the present disclosure, the electronic device (100) may display a first preview image (200) or a second preview image (300) through a display (130) and provide it to the user. The electronic device (100) may display a first captured image obtained by capturing the first preview image (200) or a second captured image obtained by capturing the second preview image (300) through a display (130) and provide it to the user.

[0071] Additionally, the electronic device (100) can provide to the user by displaying an alarm (400) through a display (130) that includes time information at which the shadow (330) of a sub-object (320) is predicted to overlap with the main object area (220) or time information at which the overlap between the shadow (330) of a sub-object (320) and the main object area (220) is predicted to end.

[0072] In one embodiment of the present disclosure, the display (130) may include any one of a liquid crystal display, a plasma display, an organic light emitting diode display, or an inorganic light emitting diode display. However, the present disclosure is not limited thereto, and the display (130) may include other types of displays capable of displaying at least one of a first preview image (200), a second preview image (300), a first captured image, a second captured image, or an alarm (400).

[0073] In one embodiment of the present disclosure, the first camera (110) and the second camera (120) may each include an RGB camera capable of acquiring an image containing RGB information. However, the present disclosure is not limited thereto, and the first camera (110) and the second camera (120) may each include an RGB-Depth camera that acquires an image containing RGB information and depth information, or a monochrome camera that acquires a monochrome image, and are not limited to any one of these.

[0074] In one embodiment of the present disclosure, at least one processor (150) controls a first camera (110) and a second camera (120) to capture a specific scene, so that an electronic device (100) can acquire a first preview image (200) and a second preview image (300). The electronic device (100) can acquire a first captured image by capturing the first preview image (200) through the first camera (110). The electronic device (100) can acquire a second captured image by capturing the second preview image (300) through the second camera (120).

[0075] In one embodiment of the present disclosure, with reference to FIG. 1, the first camera (110) and the second camera (120) are shown arranged in a vertical line on the electronic device (100) and having the same size, but the present disclosure is not limited thereto.

[0076] In one embodiment of the present disclosure, the arrangement of the first camera (110) and the second camera (120) on the electronic device (100) may be changed. Of course, the first camera (110) and the second camera (120) may be arranged diagonally or arranged in a line in a horizontal direction.

[0077] In one embodiment of the present disclosure, the sizes of the first camera (110) and the second camera (120) may differ from each other. The lens size of the second camera (120) may be smaller than the lens size of the first camera (110).

[0078] Additionally, the electronic device (100) may include three or more cameras. Of course, the electronic device (100) may include three or more cameras with different viewing angles.

[0079] Additionally, the present disclosure is not limited thereto, and the first preview image (200) and the second preview image (300) may be stored in memory (140).

[0080] In one embodiment of the present disclosure, the memory (140) may store instructions, data structures, and program code that can be read by at least one processor (150). In one embodiment of the present disclosure, the memory (140) may be one or more. Operations performed by the electronic device (100) may be implemented by at least one processor (150) executing the instructions or code of a program stored in the memory (140).

[0081] In one embodiment of the present disclosure, the memory (140) may include at least one of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), Mask ROM, Flash ROM, etc.), a hard disk drive (HDD), or a solid-state drive (SSD).

[0082] In one embodiment of the present disclosure, the memory (140) may not exist separately and may be configured to be included in at least one processor (150).

[0083] In one embodiment of the present disclosure, instructions or program code for performing functions or operations of an electronic device (100) may be stored in the memory (140). The instructions, algorithms, data structures, program code, and application programs stored in the memory (140) may be implemented in a programming or scripting language such as, for example, C, C++, Java, Python, assembler, etc.

[0084] In one embodiment of the present disclosure, various types of modules that can be used to perform the operation of the electronic device (100) may be stored in the memory (140).

[0085] In one embodiment of the present disclosure, the memory (140) may store an object detection module (141), a light source identification module (142), and an overlap determination module (143). However, not all of the modules illustrated in FIG. 2 are required. More modules than those illustrated in FIG. 2 may be stored in the memory (140), or fewer modules may be stored.

[0086] In one embodiment of the present disclosure, a 'module' included in the memory (140) may mean a unit that processes a function or operation performed by at least one processor (150). The 'module' included in the memory (140) may be implemented as software such as instructions, algorithms, data structures, or program code.

[0087] In one embodiment of the present disclosure, the object detection module (141) may be composed of instructions or program code regarding operations or functions for detecting or segmenting objects included in an image.

[0088] In one embodiment of the present disclosure, the object detection module (141) may include an algorithm for detecting or segmenting objects included in an image. The object detection module (141) may include a pre-trained artificial intelligence model for detecting or segmenting objects included in an image.

[0089] In one embodiment of the present disclosure, by having at least one processor (150) execute instructions, program code, or algorithms of an object detection module (141), the electronic device (100) can detect a main object (210) included in a first preview image (200). Additionally, the electronic device (100) can detect a sub-object (320) included in a second preview image (300).

[0090] In one embodiment of the present disclosure, the light source identification module (142) may be composed of instructions or program code regarding operations or functions for identifying information about a light source located in a space containing an electronic device (100) based on the position of a sub-object (320), the height of a sub-object (320), and the length of a shadow cast by a sub-object (320).

[0091] In one embodiment of the present disclosure, at least one processor (150) can obtain information about a light source located in a space containing an electronic device (100) based on the location of a sub-object (320), the height of a sub-object (320), and the length of a shadow cast by a sub-object (320), by executing an instruction, program code, or algorithm of a light source identification module (142).

[0092] Hereinafter, the operation by the light source identification module (142) will be described later in FIGS. 4 to 10.

[0093] In one embodiment of the present disclosure, the overlap determination module (143) may be composed of commands or program code regarding an operation or function for determining whether the shadow (330) caused by the main object area (220) and the sub object (320) included in the first preview image (200) overlaps.

[0094] In one embodiment of the present disclosure, the overlap determination module (143) may be composed of instructions or program code regarding an operation or function for calculating an overlap prediction time, which is the time taken until the overlap start frame at which the shadow of a sub-object is predicted to begin overlapping with the main object area.

[0095] In one embodiment of the present disclosure, the overlap determination module (143) may be composed of instructions or program code regarding an operation or function for calculating an overlap end time, which is the time taken from an overlap start frame in which the shadow of a sub-object is predicted to begin overlapping with the main object area to an overlap end frame in which the overlapping of the shadow of a sub-object with the main object area is predicted to end.

[0096] In one embodiment of the present disclosure, at least one processor (150) can determine whether there is an overlap between a main object area (220) and a sub-object (320) and a shadow (330) by executing an instruction, program code, or algorithm of an overlap determination module (143). Additionally, at least one processor (150) can calculate and obtain an overlap prediction time or an overlap end time by executing an instruction, program code, or algorithm of an overlap determination module (143).

[0097] Hereinafter, the operation by the overlap judgment module (143) will be described later in FIGS. 11 to 15.

[0098] In one embodiment of the present disclosure, at least one processor (150) may be configured to control a series of processes to operate an electronic device (100) according to the embodiments described below, and may be composed of one or more processors.

[0099] In one embodiment of the present disclosure, at least one processor (150) may be composed of at least one of a Central Processing Unit, a microprocessor, a Graphic Processing Unit, an Application Processor (AP), an Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), a Communication Processor (CP), a Neural Processing Unit, or an AI-dedicated processor designed with a hardware structure specialized for the learning and processing of an artificial intelligence model (AI), but is not limited thereto.

[0100] In one embodiment of the present disclosure, if one or more processors included in at least one processor (150) are artificial intelligence dedicated processors, said artificial intelligence dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0101] In one embodiment of the present disclosure, at least one processor (150) may be composed of circuitry such as a System on Chip (SoC) or an Integrated Circuit (IC). In one embodiment of the present disclosure, at least one processor (150) may include processing circuitry.

[0102] In one embodiment of the present disclosure, at least one processor (150) can execute various types of modules stored in memory (140). At least one processor (150) can execute at least one instruction constituting various types of modules stored in memory (140). By executing a program or at least one instruction stored in memory (140), at least one processor (150) can process data according to a predefined operation rule or artificial intelligence model.

[0103] In one embodiment of the present disclosure, at least one processor (150) may include a plurality of processors. In one embodiment of the present disclosure, at least one of a plurality of modules in memory (140) may be executed by any one of the plurality of processors. The remaining modules among the plurality of modules stored in memory (140) may be executed by another of the plurality of processors.

[0104] In one embodiment of the present disclosure, the inertial measuring device (160) may include at least one of an accelerometer, a gyroscope, or a magnetometer. The inertial measuring device (160) may be a sensor configured to measure at least one of the moving speed, direction, angle, or gravitational acceleration of an electronic device (100) through at least one of an accelerometer, a gyroscope, or a magnetometer.

[0105] In one embodiment of the present disclosure, when a user moves the electronic device (100), at least one processor (150) can measure at least one of the movement speed, direction, angle, or gravitational acceleration of the electronic device (100) through an inertial measuring device (160).

[0106] In one embodiment of the present disclosure, the electronic device (100) may obtain a 3-degree-of-freedom (3 DoF) measurement value including three-axis angular velocity values ​​(roll, yaw, and pitch) by using an inertial measurement device (160). Additionally, the electronic device (100) may obtain a 6-degree-of-freedom (6 DoF) measurement value including a user's three-dimensional position coordinate values ​​(x-axis, y-axis, and z-axis coordinate values) and three-axis angular velocity values ​​(roll, yaw, and pitch) by using an inertial measurement device (160).

[0107] In one embodiment of the present disclosure, a GPS device (170) receives a satellite signal from a satellite and can measure the position, speed, etc. of an electronic device (100) based on the received satellite signal. In one embodiment of the present disclosure, at least one processor (150) controls the GPS device (170), so that the electronic device (100) can obtain the height and direction of the sun based on the time obtained through the GPS device (170) and the position information of the electronic device (100).

[0108] In one embodiment of the present disclosure, the input / output interface (160) may include an HDMI port (High-Definition Multimedia Interface port), a DVI (Digital Visual Interface), a component jack, a PC port, or a USB port (Universal Serial Bus port). However, the present disclosure is not limited to the above-mentioned input / output methods.

[0109] In one embodiment of the present disclosure, at least one processor (150) controls an input / output interface (180), so that the electronic device (100) can perform input / output operations with an external electronic device, etc. through the input / output interface (180).

[0110] In one embodiment of the present disclosure, the communication interface (170) may include, for example, a wired LAN, wireless LAN, Wi-Fi, Bluetooth, Zigbee, WFD (Wi-Fi Direct), infrared communication (IrDA, infrared Data Association), BLE (Bluetooth Low Energy), NFC (Near Field Communication), Wibro (Wireless Broadband Internet), WiMAX (World Interoperability for Microwave Access), SWAP (Shared Wireless Access Protocol), WiGig (Wireless Gigabit Alliance), and RF communication.

[0111] In one embodiment of the present disclosure, by controlling a communication interface (170) with at least one processor (150), the electronic device (100) can perform data communication with an external server or an external electronic device using at least one of the communication methods included in the communication interface (170).

[0112] In one embodiment of the present disclosure, the electronic device (100) may obtain a pre-trained artificial intelligence model included in the object detection module (141) from an external server or external electronic device through a communication interface (190).

[0113] FIG. 3 is a flowchart for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0114] Referring to FIGS. 1, 2 and 3, in one embodiment of the present disclosure, a method of operating an electronic device (100) may include the step (S100) of obtaining a main object area (220) including a main object (210) and a surrounding area (230) adjacent to the main object area (220) from a first preview image (200) obtained through a first camera (110) having a first viewing angle.

[0115] In step S100, the electronic device (100) can acquire a first preview image (200) through a first camera (110). In step S100, the electronic device (100) can detect a main object (210) from the first preview image (200) through an object detection module (141) and divide the first preview image (200) into a main object area (220) and a surrounding area (230) adjacent to the main object area (220).

[0116] In one embodiment of the present disclosure, a method of operation of an electronic device (100) may include a step (S200) of detecting a sub-object (320) that is included in an area different from the main object area (220) and the surrounding area (230) and moves across a plurality of frames from a second preview image (300) obtained through a second camera (120) having a second angle of view wider than the first angle of view.

[0117] In step S200, the electronic device (100) can acquire a second preview image (300) through a second camera (120). The electronic device (100) can detect a sub-object (320) that is included in an area different from the main object area (220) and the surrounding area (230) and moves across multiple frames from the second preview image (200) through an object detection module (141).

[0118] In one embodiment of the present disclosure, the method of operation of the electronic device (100) may include the step (S300) of obtaining light source information regarding the position of a light source in a space containing the electronic device (100) based on the position of a sub-object (320), the height of the sub-object (320), and the length of a shadow (330) cast by the sub-object (320).

[0119] In step S300, the electronic device (100) can obtain light source information regarding the location of a light source in a space containing the electronic device (100) based on the location of a sub-object (320), the height of the sub-object (320), and the length of the shadow (330) cast by the sub-object (320) through a light source identification module (142).

[0120] Hereinafter, the operation of the electronic device (100) in step S300 will be described later in FIGS. 4 to 10.

[0121] In one embodiment of the present disclosure, the method of operation of the electronic device (100) may include the step (S400) of providing time information for a frame in which the shadow of a sub-object (320) overlaps with a main object area (220), obtained based on light source information, the position of the sub-object (320), and the height of the sub-object (320).

[0122] In step S400, the electronic device (100) can obtain time information for a frame in which the shadow of a sub-object (320) overlaps with a main object area (220), obtained based on light source information, the position of the sub-object (320), and the height of the sub-object (320) through an overlap determination module (143), and provide the obtained time information to a user.

[0123] Hereinafter, the operation by step S400 will be described in FIGS. 11 to 15.

[0124] FIG. 4 is a flowchart illustrating an operation to acquire two-dimensional shadow information in order to acquire the length of a shadow cast by a sub-object according to an embodiment of the present disclosure. FIG. 5 is a diagram illustrating an operation to acquire two-dimensional shadow information in order to acquire the length of a shadow cast by a sub-object according to an embodiment of the present disclosure. Hereinafter, the same reference numerals are assigned to steps identical to those described in FIG. 3, and redundant descriptions are omitted.

[0125] Referring to FIGS. 2, 3 and 4, in one embodiment of the present disclosure, the step of detecting a sub-object (S200) may include the step (S210) of acquiring a plurality of second preview images each comprising a sub-object (320) whose position is changed over a plurality of frames through a second camera (120).

[0126] In step S210, the electronic device (100) can acquire a plurality of second preview images corresponding to each of the plurality of frames through the second camera (120) over a plurality of frames. At this time, as the sub-object (320) moves during the plurality of frames, the position of the sub-object (320) may change. Accordingly, the position of the sub-object (320) included in each of the plurality of second preview images may change.

[0127] In one embodiment of the present disclosure, step S210 may be performed after step S100.

[0128] In one embodiment of the present disclosure, the step of acquiring light source information (S300) may include the step (S311) of extracting a sub-object (320) and a shadow (330) of the sub-object (320) from each of a plurality of second preview images.

[0129] Referring to FIG. 4 and FIG. 5, in one embodiment of the present disclosure, FIG. 5 shows a second preview image (300) of any one of a plurality of second preview images.

[0130] In one embodiment of the present disclosure, an electronic device (100) may extract a sub-object (320) and a shadow (330) of a sub-object (320) from one of the second preview images (300) using a thresholding technique. At this time, the extracted sub-object (320) and the shadow (330) of the sub-object (320) may refer to a plurality of pixels (340) constituting the sub-object (320) and the shadow (330) of the sub-object (320).

[0131] In step S300, the electronic device (100) can repeat the above-described operation for each of the plurality of second preview images to extract a sub-object (320) and a shadow (330) of the sub-object (320) from each of the plurality of second preview images.

[0132] In one embodiment of the present disclosure, the step of acquiring light source information (S300) may include the step (S312) of dividing the sub-object (320) and the shadow (330) of the sub-object (320) from the extracted sub-object (320) and the shadow (330) of the sub-object (320), respectively.

[0133] In step S312, the electronic device (100) can divide the sub-object (320) and the shadow (330) of the sub-object (320) from a plurality of pixels (340) constituting the sub-object (320) and the shadow (330) of the sub-object (320) through the object detection module (141).

[0134] In one embodiment of the present disclosure, the step of acquiring light source information (S300) may include the step (S313) of acquiring two-dimensional shadow information for the shadow (330) of the sub-object (320) included in each of the plurality of second preview images by removing the divided sub-object (320) from the extracted sub-object (320) and the shadow (330) of the sub-object (320).

[0135] In step S313, the electronic device (100) can obtain two-dimensional shadow information for the shadow (330) of the sub-object (320) included in each of the plurality of second preview images by removing the sub-object (320) from the extracted sub-object (320) and the shadow (330) of the sub-object (320). At this time, the two-dimensional shadow information may mean information about the remaining pixels corresponding to the shadow (330) by the sub-object (320) from which the sub-object (320) has been removed among the plurality of pixels (340).

[0136] However, the present disclosure is not limited thereto, and the operations in steps S311 to S313 may be performed in a single step. The electronic device (100) may acquire the sub-object (320) and the shadow (330) of the sub-object (320) each from any one second preview image (300) by using a first threshold value for extracting the sub-object (320) and a second threshold value smaller than the first threshold value for extracting the shadow (330) of the sub-object (320). Additionally, the electronic device (100) may acquire the sub-object (320) and the shadow (330) of the sub-object (320) each from any one second preview image (300) through an object detection module (141).

[0137] In one embodiment of the present disclosure, the step of obtaining light source information (S300) may include the step (S314) of obtaining the length of the shadow (330) by the sub-object (320) based on two-dimensional shadow information.

[0138] In step S314, the electronic device (100) can obtain the length of the shadow (330) using pixels corresponding to the shadow (330) by the sub-object (320) based on two-dimensional shadow information.

[0139] FIG. 6 is a flowchart illustrating an operation to acquire light source information based on three-dimensional position information of a sub-object, the height of the sub-object, and the length of a shadow according to an embodiment of the present disclosure. Hereinafter, the same reference numerals are assigned to steps identical to those described in FIG. 3 and FIG. 4, and redundant descriptions are omitted.

[0140] Referring to FIGS. 2, FIGS. 3 and FIGS. 6, in one embodiment of the present disclosure, the step of acquiring light source information (S300) may include the step (S320) of acquiring three-dimensional sub-position information of a sub-object (320) using depth of field (DOF) from a plurality of second preview images.

[0141] In step S320, the electronic device (100) can obtain three-dimensional sub-location information of a sub-object (320) using depth from a plurality of second preview images through a light source identification module (142).

[0142] The operation of step S320 will be described below in Fig. 7.

[0143] In one embodiment of the present disclosure, the step of acquiring light source information (S300) may include the step (S330) of acquiring a plurality of top-view images by inverse mapping a plurality of second preview images.

[0144] In step S330, the electronic device (100) can inverse map a plurality of second preview images through a light source identification module (142) to obtain a plurality of top-view images corresponding to each of the plurality of second preview images.

[0145] In one embodiment of the present disclosure, steps S320 and S330 may be performed after step S210.

[0146] In one embodiment of the present disclosure, the step of acquiring light source information (S300) may include the step (S331) of acquiring top surface position information including the distance traveled by a sub-object (320) and the position of the sub-object (320) during a plurality of frames based on a plurality of top surface views.

[0147] In step S331, the electronic device (100) can obtain top surface position information including the distance traveled by the sub-object (320) and the position of the sub-object (320) during a plurality of frames, based on a plurality of top surface views, through a light source identification module (142).

[0148] The operation of steps S330 and S331 will be described below in FIG. 8.

[0149] In one embodiment of the present disclosure, the step of acquiring light source information (S300) may include the step (S340) of acquiring three-dimensional position information of a sub-object (320), including the position of the sub-object (320) and the height of the upper surface object (320), based on sub-position information and upper surface position information.

[0150] In step S340, the electronic device (100) can obtain three-dimensional position information of a sub-object (320), including the position of the sub-object (320) and the height of the upper surface object (320), based on sub-position information and upper surface position information, through a light source identification module (142).

[0151] In one embodiment of the present disclosure, the step of acquiring light source information (S300) may include the step (S350) of acquiring the length of the shadow (330) for the sub-object (320) from a plurality of top views based on the two-dimensional shadow information of the sub-object (320) acquired in step S313 and the three-dimensional position information of the sub-object (320) acquired in step S340.

[0152] In step S350, the electronic device (100) can obtain the length of the shadow (330) for the sub-object (320) from a plurality of top views based on the two-dimensional shadow information of the sub-object (320) and the three-dimensional position information of the sub-object (320) through the light source identification module (142).

[0153] The operation of steps S340 and S350 will be described later in FIG. 9.

[0154] In one embodiment of the present disclosure, the step of acquiring light source information (S300) may include the step (S360) of acquiring light source information for a light source located in a space containing an electronic device (100), based on the three-dimensional position information of a sub-object (320) acquired in step S340 and the length of a shadow (330) acquired in step S350.

[0155] In step S360, the electronic device (100) can obtain light source information for a light source located in the space containing the electronic device (100) based on the three-dimensional position information of the sub-object (320) and the length of the shadow (330) through the light source identification module (142). At this time, the light source information may include position information of the light source.

[0156] The operation of step S360 will be described below in FIG. 10.

[0157] In one embodiment of the present disclosure, the operation of step S400 may be performed after the operation of step S360.

[0158] FIG. 7 is a diagram illustrating an operation to obtain three-dimensional first sub-position information of a sub-object using depth according to one embodiment of the present disclosure.

[0159] Referring to FIG. 2, FIG. 6 and FIG. 7, in one embodiment of the present disclosure, FIG. 7 shows the camera position (710) of the second camera (120) and the sub-object (320) in order to obtain three-dimensional sub-position information of the sub-object (320) using the depth of field (DOF).

[0160] In this context, "depth of field" refers to the range in which an object is clearly recognized when in focus during shooting with a camera, and can mean the range in which an object within a certain distance from the focal plane of the lens included in the camera appears clearly.

[0161] In one embodiment of the present disclosure, the coordinates of the reference origin may be set to (0, 0, 0) in a three-dimensional space consisting of an x-axis, a y-axis, and a z-axis. The camera position (710) may be set to be located at (0, 0, h), where h may represent the height of the second camera (120).

[0162] In one embodiment of the present disclosure, the electronic device (100) can obtain sub-position information of a sub-object (320) by using depth to calculate the distance of the sub-object (320) from the camera position (710) in the second preview image (700). At this time, the sub-position information may include a first sub-position (720), which is the position of the sub-object (320) in a plane formed by the x-axis and the y-axis, and a first height (730), which is the height of the sub-object (320) in the z-axis. The first sub-position (720) may include a first x-axis coordinate (740) and a first y-axis coordinate (750). The first height (730) may include a first z-axis coordinate (760).

[0163] In one embodiment of the present disclosure, the position of the sub-object (320) may mean the position of the sub-object (320) in a plane formed by the x-axis and the y-axis. The height of the sub-object (320) may mean the height of the sub-object (320) in the z-axis from the position of the sub-object (320).

[0164] However, the present disclosure is not limited thereto, and the electronic device (100) may further include a Time of Flight (ToF) sensor. The electronic device (100) may also obtain sub-location information based on the distance between the sub-object (320) and the second camera (120) and the height of the sub-object (320), which are sensed through the ToF sensor.

[0165] In one embodiment of the present disclosure, the electronic device (100) can repeat the operation of FIG. 7 to obtain sub-position information of a sub-object (320) whose position is changed and which is included in each of a plurality of second preview images corresponding to each of a plurality of frames.

[0166] In one embodiment of the present disclosure, when a sub-object (320) included in a second preview image (700) moves from a first position to a second position across a plurality of frames, the sub-object located at the first position may be referred to as the first sub-object, and the sub-object located at the second position may be referred to as the second sub-object.

[0167] In one embodiment of the present disclosure, the electronic device (100) can obtain a rate of change of a sub-object based on first sub-position information of a first sub-object and second sub-position information of a second sub-object. The electronic device (100) can obtain the ratio of a first x-axis coordinate (740) included in the sub-position information and a second x-axis coordinate included in the second sub-position information as the rate of change of the sub-object.

[0168] In one embodiment of the present disclosure, the electronic device (100) may obtain a first movement distance in the x-axis direction toward the camera position (710) of a sub-object that has moved over a plurality of frames, based on sub-position information of each of the first sub-object and the second sub-object. In one embodiment of the present disclosure, the sub-position information may include the rate of change of the sub-object and the first movement distance.

[0169] FIG. 8 is a diagram illustrating the operation of obtaining three-dimensional second sub-position information of a sub-object using a top view obtained through inverse mapping according to one embodiment of the present disclosure.

[0170] Referring to FIG. 2, FIG. 6 and FIG. 8, in one embodiment of the present disclosure, FIG. 8 shows a second preview image (800) among a plurality of second preview images and a top view (830) obtained by inverse mapping the second preview image (800).

[0171] In one embodiment of the present disclosure, the second preview image (800) includes a first sub-object (320) located at a first position and a second sub-object (321) located at a second position to represent a sub-object moving across a plurality of frames. The first sub-object (320) and the second sub-object (321) may be the same sub-object, and their positions may change over time as they move.

[0172] In one embodiment of the present disclosure, the electronic device (100) can obtain a first sub-position (720) of a first sub-object (320) and a first height (730) of the first sub-object (320) using depth. The electronic device (100) can obtain a second sub-position (810) of a second sub-object (321) and a first height (820) of the second sub-object (321) using depth.

[0173] In one embodiment of the present disclosure, the electronic device (100) can obtain a top view (830) by inverse mapping a second preview image (800). At this time, "inverse mapping" means transforming the coordinate system of the image and mapping the coordinates of the transformed image to the coordinates of the original image.

[0174] In one embodiment of the present disclosure, the electronic device (100) may obtain an inverse mapping function through coordinate mapping between a second preview image (800) and a top view (830) based on parameters such as the focal length of the second camera (120), the size and position of the lens, etc. The electronic device (100) may obtain a top view (830) by transforming the second preview image (800) through the inverse mapping function. At this time, the top view (830) may refer to an image viewed from the z-axis.

[0175] However, the present disclosure is not limited thereto, and the electronic device (100) may obtain a top view (830) by converting a second preview image (800) through an artificial intelligence model such as NeRF (Neural Radiance Fields), cGAN (Conditional Generative Adversarial Network), or VAE (Variational AutoEncoder).

[0176] In one embodiment of the present disclosure, the top view (830) may include a first top view position (840) corresponding to the camera position (710) of the second camera and the first sub-position (720) of the first sub-object (320). The top view (830) may include a second top view position (850) corresponding to the second sub-position (810) of the second sub-object (321).

[0177] In one embodiment of the present disclosure, the electronic device (100) can obtain top surface position information of the first sub-object (320) by calculating the distance of the first sub-object (320) from the camera position (710) according to the top surface view (830). At this time, the top surface position information of the first sub-object (320) may include a first top surface position (840), which is the position of the first sub-object (320) in a plane formed by the x-axis and the y-axis.

[0178] In one embodiment of the present disclosure, the electronic device (100) can repeat the operation of FIG. 8 to obtain upper surface position information of a sub-object that is included in each of a plurality of second preview images corresponding to each of a plurality of frames and whose position is changed.

[0179] In one embodiment of the present disclosure, the electronic device (100) can obtain a second movement distance (850) in which a sub-object that has moved across a plurality of frames moves in the x-axis direction toward a camera position (710), based on the upper surface position information of each of the first sub-object (320) and the second sub-object (321).

[0180] In one embodiment of the present disclosure, the upper surface position information may include a second travel distance (850).

[0181] FIG. 9 is a diagram illustrating an operation to obtain the length of a shadow by a sub-object based on two-dimensional shadow information and three-dimensional position information according to one embodiment of the present disclosure.

[0182] Referring to FIGS. 2, 7, 8 and 9, in one embodiment of the present disclosure, an electronic device (100) can obtain the length of a first shadow (950) of a first sub-object (320) and the length of a second shadow (960) of a second sub-object (321) from a top view (900). At this time, the length of the first shadow (950) can be obtained based on three-dimensional position information of the first sub-object (320) and two-dimensional shadow information of the first sub-object (320). The length of the second shadow (960) can be obtained based on three-dimensional position information of the second sub-object (321) and two-dimensional shadow information of the second sub-object (321).

[0183] In one embodiment of the present disclosure, three-dimensional position information of the first sub-object (320) may be obtained based on the sub-position information of the first sub-object (320) and the top surface position information of the first sub-object (320). Three-dimensional position information of the second sub-object (321) may be obtained based on the sub-position information of the second sub-object (321) and the top surface position information of the second sub-object (321).

[0184] In one embodiment of the present disclosure, the electronic device (100) may obtain the average of a first travel distance included in first sub-location information and a second travel distance (850) included in second sub-location information as a reference travel distance.

[0185] In one embodiment of the present disclosure, the electronic device (100) may set condition 1 such that the first position (910) of the first sub-object (320) is located between the first sub-position (720) of the first sub-object (320) and the first upper position (840), and the second position (920) of the second sub-object (321) is located between the second sub-position of the second sub-object (321) and the second upper position (850).

[0186] In one embodiment of the present disclosure, the electronic device (100) may set condition 2 such that the difference between the first distance and the second distance is equal to the reference distance, where the distance in the x-axis direction between the first position (910) and the camera position (710) is called the first distance, and the distance in the x-axis direction between the second position (920) and the camera position (710) is called the second distance.

[0187] In one embodiment of the present disclosure, the electronic device (100) may set a condition 3 such that the difference between the ratio between a first distance and a second distance and the rate of change of a sub-object included in the first sub-location information is minimized.

[0188] In one embodiment of the present disclosure, the electronic device (100) may obtain a first position (910) and a second position (920) satisfying conditions 1 to 3 as three-dimensional position information of a first sub-object (320) and three-dimensional position information of a second sub-object (321), respectively.

[0189] In one embodiment of the present disclosure, the electronic device (100) can obtain a first position (910) and a second position (920) by using an optimization algorithm, for example, a Lagrangian Function, with condition 3 as the objective function and conditions 1 and 2 as constraints.

[0190] Through this, the electronic device (100) of the present disclosure can obtain three-dimensional position information of a sub-object with high accuracy based on depth and top view from a second preview image (300).

[0191] In one embodiment of the present disclosure, the three-dimensional position information of the first sub-object (320) may include a first position (910) and a first height (730). The three-dimensional position information of the second sub-object (321) may include a second position (920) and a second height (820).

[0192] In one embodiment of the present disclosure, the electronic device (100) can obtain the distance (950) between the first position (910) included in the three-dimensional position information of the first sub-object (320) and the first top pixel (930) that is furthest in the y-axis direction among the pixels corresponding to the shadow included in the two-dimensional shadow information of the first sub-object (320) as the length (950) of the first shadow by the first sub-object (320).

[0193] In one embodiment of the present disclosure, the electronic device (100) can obtain the distance (960) between the second position (920) included in the three-dimensional position information of the second sub-object (321) and the second top pixel (940) that is furthest in the y-axis direction among the pixels corresponding to the shadow included in the two-dimensional shadow information of the second sub-object (321) as the length (960) of the second shadow by the second sub-object (321).

[0194] FIG. 10 is a diagram illustrating an operation to acquire light source information based on three-dimensional position information of a sub-object, the height of the sub-object, and the length of the shadow according to one embodiment of the present disclosure.

[0195] Referring to FIG. 2, FIG. 6 and FIG. 10, in one embodiment of the present disclosure, FIG. 10 shows an electronic device (100), a light source (1000) in a space containing the electronic device (100), a sub-object (320) that moves across a plurality of frames and has its position changed, and a shadow (330) by the sub-object (320).

[0196] In one embodiment of the present disclosure, the light source (1000) may vary depending on the type of space where the electronic device (100) is located, such as sunlight or lighting.

[0197] In one embodiment of the present disclosure, the electronic device (100) can obtain three-dimensional position information of a sub-object (320) across a plurality of frames through a light source identification module (142). Additionally, the electronic device (100) can obtain the length of the shadow cast by the sub-object (320) by using the three-dimensional position information of the sub-object (320) and the upper pixels included in the two-dimensional shadow information cast by the sub-object (320) through the light source identification module (142).

[0198] In one embodiment of the present disclosure, FIG. 10 illustrates three-dimensional position information and top pixels of sub-objects (320) located at three different positions as they move across a plurality of frames.

[0199] In one embodiment of the present disclosure, the electronic device (100) can obtain a first position (1010) and a first height (1011) of a sub-object (320) in a first frame. Additionally, the electronic device (100) can obtain a first top pixel (1012) of a shadow in a first frame and obtain the length of the shadow cast by the sub-object (320) in the first frame.

[0200] In one embodiment of the present disclosure, the electronic device (100) can obtain a second position (1020) and a second height (1021) of a sub-object (320) in a second frame. Additionally, the electronic device (100) can obtain a second top pixel (1022) of a shadow in a second frame and obtain the length of the shadow cast by the sub-object (320) in the second frame.

[0201] In one embodiment of the present disclosure, the electronic device (100) can obtain a third position (1030) and a third height (1031) of a sub-object (320) in a third frame. Additionally, the electronic device (100) can obtain a third top pixel (1032) of the shadow in the third frame and obtain the length of the shadow cast by the sub-object (320) in the third frame.

[0202] In one embodiment of the present disclosure, an electronic device (100) can calculate a first line passing through a first height (1011) and a first top pixel (1012), a second line passing through a second height (1021) and a second top pixel (1022), and a third line passing through a third height (1031) and a third top pixel (1032). The electronic device (100) can obtain light source information that a light source (1000) is located at the point where the first line, the second line, and the third line intersect.

[0203] In one embodiment of the present disclosure, the position of the light source (1000) may be obtained as a point where the first line, the second line, and the third line intersect. However, the present disclosure is not limited thereto, and the electronic device (100) may obtain light source information that the position of the light source (1000) is centered at the point where at least two of the first line, the second line, and the third line intersect, and the light source (1000) is located in an area having a radius equal to a preset reference distance.

[0204] However, the present disclosure is not limited thereto. If the space containing the electronic device (100) is outdoors and the light source (1000) corresponds to the sun, the electronic device (100) may obtain light source information regarding the position of the sun based on an inertial measurement device (160) and a GPS device (170).

[0205] In one embodiment of the present disclosure, the electronic device (100) can obtain location information of the space containing the electronic device (100), such as latitude, longitude, and altitude, through a GPS device (170).

[0206] In one embodiment of the present disclosure, the electronic device (100) may obtain time information through a communication interface (190) or a GPS device (170). Based on the time information, the electronic device (100) may obtain the position of the sun, for example, the solar azimuth angle and the solar altitude angle. The electronic device (100) may also obtain information about the position of the sun based on the time information and spatial position information.

[0207] Additionally, in one embodiment of the present disclosure, the electronic device (100) may convert the Red, Green, and Blue components of each of a plurality of pixels included in at least one of the first preview image (200) or the second preview image (300) into a luminance component (Y) and a color difference component (Cb, Cr). Based on the luminance component, the electronic device (100) may identify an area having the brightest luminance component in the first preview image (200) or the second preview image (300), and may obtain light source information based on the location, shape, area, etc. of the identified area.

[0208] FIG. 11 is a diagram illustrating an operation for obtaining an overlap prediction time when a sub-object in the current frame does not overlap with the main object area, according to an embodiment of the present disclosure. Hereinafter, the same reference numerals are assigned to steps identical to those described in FIG. 3, and redundant descriptions are omitted.

[0209] Referring to FIGS. 1, FIGS. 2, FIGS. 3 and FIGS. 11, in one embodiment of the present disclosure, a method of operating an electronic device (100) may include a step (S220) of identifying whether a sub-object (320) in the current frame overlaps with a main object area (220).

[0210] In one embodiment of the present disclosure, the operation of step S220 may be performed after step S200.

[0211] In step S220, the electronic device (100) can identify, through the object detection module (141), whether a sub-object (320) overlaps with the main object area (220) in the current frame. In one embodiment of the present disclosure, the sub-object (320) is located in an area different from the main object area (220) and the surrounding area (330), but a part of the sub-object (320) may overlap with the main object area (220). In this case, the electronic device (100) can identify that the sub-object (320) overlaps with the main object area (220).

[0212] In one embodiment of the present disclosure, the step of providing time information (S400) may include the step of providing time information by including the time taken until the overlap start frame in which the shadow (330) of the sub-object (320) is predicted to start overlapping with the main object area (220) in the current frame, as the time information is provided as the overlap prediction time, as it is identified in step S220 that the sub-object (320) in the current frame does not overlap with the main object area (220).

[0213] In one embodiment of the present disclosure, the electronic device (100) may provide time information including the time taken until the overlap start frame in which the shadow (330) of the sub-object (320) in the current frame is predicted to begin overlapping with the main object area (220), as identified that the sub-object (320) in the current frame does not overlap with the main object area (220).

[0214] In one embodiment of the present disclosure, the step of providing the overlap prediction time by including it in the time information may be performed after step S300.

[0215] In one embodiment of the present disclosure, the step of providing the overlap prediction time by including it in the time information may include the step (S410) of obtaining the average speed of a sub-object (320) moving across a plurality of frames.

[0216] In step S410, the electronic device (100) can obtain the average speed of a sub-object (320) moving across multiple frames through the overlap determination module (143).

[0217] In one embodiment of the present disclosure, the step of providing the overlap prediction time including time information may include the step (S420) of obtaining the current position of the sub-object (320) in the current frame and the overlap position of the sub-object (320) in the overlap start frame.

[0218] In step S420, the electronic device (100) can obtain the current position of the sub-object (320) in the current frame and the overlap position of the sub-object (320) in the overlap start frame through the overlap determination module (143).

[0219] In one embodiment of the present disclosure, the step of providing the overlap prediction time by including it in the time information may include the step (S430) of obtaining a value obtained by dividing the difference between the overlap location and the current location by the average speed as the overlap prediction time.

[0220] In step S430, the electronic device (100) can obtain a value obtained by dividing the difference between the overlap location and the current location by the average speed through the overlap determination module (143) as the overlap prediction time.

[0221] In one embodiment of the present disclosure, when the value obtained by dividing the difference between the overlap location and the current location by the average speed is 5 seconds, the electronic device (100) may provide to the user information of the overlap prediction time, including the information that "the main object area (220) may be obscured by the shadow (330) of the sub-object (320) after 5 seconds."

[0222] Hereinafter, the operation of the electronic device (100) in steps S410 to S420 will be described later in FIG. 12.

[0223] In one embodiment of the present disclosure, the electronic device (100) may identify, based on the result obtained in steps S300 to S430, that the main object area (220) is not obscured by the shadow (330) of the sub object (320).

[0224] In one embodiment of the present disclosure, the electronic device (100) may identify, based on light source information, that the shadow (330) of the sub-object (320) is generated in the opposite direction, rather than in the direction facing the main object area (220), due to the positional relationship between the position of the light source (1000) and the sub-object (320).

[0225] In one embodiment of the present disclosure, the electronic device (100) may determine, based on three-dimensional position information of the sub-object (320), that the sub-object (320) is too far from the main object area (220) so that there is no possibility of the shadow (330) of the sub-object (320) overlapping with the main object area (220). Additionally, the electronic device (100) may determine that the size of the sub-object (320) is small so that there is no possibility of the shadow (330) of the sub-object (320) overlapping with the main object area (220).

[0226] Additionally, the electronic device (100) can determine that if the direction in which the sub-object (320) moves is away from the main object area (220), there is no possibility that the shadow (330) of the sub-object (320) will overlap with the main object area (220).

[0227] In this case, the electronic device (100) may not perform the action of providing the user with time information about the frame in which the shadow (330) of the sub-object (320) overlaps with the main object area.

[0228] Additionally, in one embodiment of the present disclosure, the method of operation of the electronic device (100) may further include the step of providing time information including the time taken from an overlap start frame to a first overlap end frame, which is predicted to end when the shadow (330) of a sub-object (320) overlaps with the main object area (220).

[0229] In the step of providing the first overlap end time as time information, the electronic device (100) can calculate and obtain the time taken from the overlap start frame to the first overlap end frame, which is predicted to end when the shadow (330) of the sub-object (320) overlaps with the main object area (220), through the overlap determination module (143).

[0230] In one embodiment of the present disclosure, the method of operation of the electronic device (100) may further include the step of obtaining a first captured image by capturing a first preview image (200) through a first camera (110) after a preset shooting waiting time has elapsed.

[0231] In one embodiment of the present disclosure, the electronic device (100) may acquire a first captured image by capturing a first preview image (200) through a first camera (110) after a preset shooting waiting time. At this time, "shooting waiting time" may refer to a time set to acquire a first captured image by capturing a scene shown through the first preview image (200) after a preset time.

[0232] In one embodiment of the present disclosure, the method of operation of an electronic device (100) may further include the step of acquiring a first captured image by capturing a first preview image (200) through a first camera (110) after the time added to the shooting waiting time and the first overlap end time, based on a preset shooting waiting time and an overlap prediction time, if the overlap prediction time is shorter than the shooting waiting time.

[0233] In one embodiment of the present disclosure, when the shooting waiting time is "5 seconds," the overlap prediction time is "3 seconds," and the first overlap end time is "5 seconds," the electronic device (100) can obtain a first shooting image by shooting a first preview image (200) through the first camera (110) after "10 seconds," which is the sum of the shooting waiting time and the first overlap end time.

[0234] However, if the shooting waiting time is "5 seconds" and the overlap prediction time is "6 seconds", the main object area (220) overlaps with the shadow (330) of the sub-object (320) after the first shooting image is acquired through the first camera (110), so the electronic device (100) can acquire the first shooting image by taking the first preview image (200) after the shooting waiting time. In this case, the electronic device (100) can provide information about the end time of the first overlap to the user as time information.

[0235] In one embodiment of the present disclosure, the method of operation of an electronic device (100) may include a step (S500) of performing a time correction or frame correction operation by comparing a second overlap end time, which is the time taken from the current frame to a second overlap end frame where it is predicted that the overlap between the sub-object (320) and the main object area (220) will end, with a preset reference time, as it is identified in step S220 that the sub-object (320) overlaps with the main object area (220) in the current frame.

[0236] In step S500, as it is identified that a sub-object (320) overlaps with a main object area (220) in the current frame, the electronic device (100) can perform a time compensation or frame compensation operation by comparing a second overlap end time, which is the time taken from the current frame to a second overlap end frame where it is predicted that the overlap between the sub-object (320) and the main object area (220) will end, with a preset reference time through an overlap determination module (143).

[0237] Hereinafter, the operation of the electronic device (100) in step S500 will be described later in FIGS. 13 to 15.

[0238] FIG. 12 is a diagram illustrating an operation for obtaining an overlap prediction time according to one embodiment of the present disclosure.

[0239] Referring to FIGS. 1, FIGS. 2, FIGS. 11 and FIGS. 12, in one embodiment of the present disclosure, FIG. 12 shows a main object area (220) containing a main object, an electronic device (100) that photographs the main object area (220), a light source (1000) located in a space containing the electronic device (100), a sub-object (320) that moves over a plurality of frames and changes its position, and a shadow (330) of the sub-object (320).

[0240] In one embodiment of the present disclosure, the electronic device (100) can obtain light source information regarding the position of a light source (1000) based on three-dimensional position information of a sub-object (320) obtained over a plurality of frames and two-dimensional shadow information.

[0241] In one embodiment of the present disclosure, an electronic device (100) can obtain the average speed of a sub-object (320) whose position changes over a plurality of frames. The electronic device (100) can obtain the average speed of the sub-object (320) by dividing the position change of the sub-object (320) that changed during a plurality of frames into a plurality of frames.

[0242] However, the present disclosure is not limited thereto, and the electronic device (100) may obtain the average speed of the sub-object (320) by giving a high weight to the position change of the sub-object (320) in the most recently measured frame among the plurality of frames, and giving a low weight to the position change of the sub-object (320) as the time of measurement is older in the past frame among the plurality of frames.

[0243] In one embodiment of the present disclosure, the electronic device (100) can predict the position of the sub-object (320) in the next frame based on the average speed of the acquired sub-object (320). The electronic device (100) can move the three-dimensional position information of the sub-object (320) based on the average speed to obtain three-dimensional predicted position information where the sub-object (320) is predicted to be located.

[0244] In one embodiment of the present disclosure, the electronic device (100) can predict the location of a sub-object (320) where the shadow (330) of the sub-object (320) is predicted to begin overlapping with the main object area (220) based on three-dimensional predicted location information and light source information. A frame in which the sub-object (320) is located at the location where the shadow (330) of the sub-object (320) is predicted to begin overlapping with the main object area (220) may be referred to as the "overlapping start frame."

[0245] In one embodiment of the present disclosure, the electronic device (100) can obtain an overlap prediction time by dividing the difference between the position of the sub-object (320) in the current frame and the position of the sub-object (320) in the overlap time frame by the average speed of the sub-object (320).

[0246] In one embodiment of the present disclosure, FIG. 12 illustrates a sub-object (320) located at a first position (1200) in the current frame. Herein, "current frame" may refer to the frame at the time when the electronic device (100) acquires the first preview image (200) and the second preview image (300).

[0247] Additionally, the position of the sub-object (320) may change in subsequent frames as time progresses. FIG. 12 illustrates a sub-object (320) located at a second position (1210) in the first frame, which is the next frame, after time has passed from the current frame, and a sub-object (320) located at a third position (1220) in the second frame, which is the frame following the first frame.

[0248] In one embodiment of the present disclosure, the electronic device (100) can obtain the average speed (1260) of the sub-object (320) based on the position change of the sub-object (320) from a first position (1200) to a third position (1220) and the flow of time from the current frame to the second frame. At this time, when obtaining the average speed (1260) of the sub-object (320), the position change from the first frame to the second frame may be reflected with a greater weight applied than the position change from the current frame to the first frame.

[0249] In one embodiment of the present disclosure, the electronic device (100) can predict the position of the sub-object (320) after the third frame based on the acquired average speed (1260) and the three-dimensional position information of the sub-object (320).

[0250] In one embodiment of the present disclosure, the electronic device (100) can predict the shadow (330) of the sub-object (320) generated at a given location and the length of the shadow (330) based on the predicted location of the sub-object (320), the height of the sub-object (320) at the given location, and light source information. The electronic device (100) can obtain the location of the sub-object (320) at which the generated shadow (330) begins to overlap with the main object area (220) as the overlap location (1230). The electronic device (100) can obtain the frame at which the sub-object (320) is predicted to be located at the overlap location (1230) as the overlap start frame.

[0251] At this time, the length of the shadow (330) at the overlap location (1230) can be obtained based on the height of the sub-object (320) at the overlap location (1230) and the position of the light source (1000) included in the light source information. When the angle between the line connecting the point where the shadow (330) and the main object area (220) meet at the overlap location (1230) and the light source (1000) and the ground (a plane consisting of the x-axis and y-axis) is denoted as the light source angle P, the product of the tangent (P) among trigonometric functions and the length of the shadow (330) at the overlap location (1230) may be equal to the height of the sub-object (320).

[0252] In one embodiment of the present disclosure, an electronic device (100) can obtain an overlap prediction time by dividing the distance from a first position (1200) to an overlap position (1230) by an average speed (1260). The electronic device (100) can provide time information including the overlap prediction time to a user in the form of an alarm or the like.

[0253] Additionally, the electronic device (100) can obtain an overlap end position where it is predicted that the shadow (330) of the sub-object (320) will end to overlap with the main object area (220) as the sub-object (320) moves from the overlap position (1230) at an average speed (1260). The electronic device (100) can obtain a frame where the sub-object (320) is predicted to be located at the overlap end position as a first overlap end frame. At this time, the overlap end position may mean a position where neither the sub-object (320) nor the shadow (330) of the sub-object (320) overlaps with the main object area (220).

[0254] In one embodiment of the present disclosure, the electronic device (100) may obtain a first overlap end time by dividing the distance from the overlap position (1230) to the overlap end position by the average speed (1260). The electronic device (100) may provide time information including the first overlap end time to the user in the form of an alarm or the like.

[0255] Additionally, the electronic device (100) may control the first camera (110) to acquire a first captured image by capturing a first preview image (200) after a time equal to the first overlap end time has elapsed from a preset shooting waiting time.

[0256] FIG. 13 is a diagram illustrating an operation to perform time correction or frame correction according to whether the second overlap end time is shorter than the reference time when a sub-object overlaps with the main object area in the current frame, according to an embodiment of the present disclosure. Hereinafter, the same reference numerals are assigned to steps identical to those described in FIG. 11, and redundant descriptions are omitted.

[0257] Referring to FIGS. 1, FIGS. 2, FIGS. 11 and FIGS. 13, in one embodiment of the present disclosure, the step (S500) of performing a time correction or frame correction operation may include the step (S510) of identifying whether a second overlap end time, which is the time taken from the current frame to a second overlap end frame at which the overlap between the sub-object (320) and the main object area (220) is predicted to end, is shorter than a preset reference time.

[0258] At this time, the “reference time” may be a preset time to serve as a reference for whether to obtain a first captured image by taking a first preview image (200) through the first camera (110) after the electronic device (100) waits until the sub-object (320) does not overlap with the main object area (220).

[0259] In one embodiment of the present disclosure, step S510 may be performed as it is identified in step S220 that a sub-object (320) in the current frame overlaps with the main object area (220).

[0260] In step S510, the electronic device (100) can identify, through the overlap determination module (143), whether the second overlap end time, which is the time taken from the current frame to the second overlap end frame where the overlap between the sub-object (320) and the main object area (220) is predicted to end, is shorter than a preset reference time.

[0261] In one embodiment of the present disclosure, as it is identified in step S510 that the second overlap end time is shorter than the reference time, the method of operation of the electronic device (100) may include the step (S520) of capturing a first preview image (200) through the first camera (110) after the second overlap end time has elapsed to obtain a first captured image.

[0262] In step S520, the electronic device (100) can obtain a first captured image by taking a first preview image (200) through the first camera (110) after the second overlap end time has elapsed.

[0263] The operation of steps S510 and S520 below will be described later in FIG. 14.

[0264] In one embodiment of the present disclosure, as the second overlap end time is identified as being equal to or longer than the reference time in step S510, the method of operation of the electronic device (100) may include the step (S530) of obtaining a first sub-shot image by taking a first preview image (200) using a first camera when the sub-object (320) is located at a first position in the main object area (220).

[0265] In step S530, the electronic device (100) can obtain a first sub-shot image by using a first camera to take a first preview image (200) when the sub-object (320) is located at a first position in the main object area (220).

[0266] In one embodiment of the present disclosure, the method of operation of the electronic device (100) may include the step (S540) of capturing a first preview image (200) using a first camera and obtaining a second sub-capture image when the sub-object (320) is located at a second position different from the first position in the main object area (220).

[0267] In step S540, the electronic device (100) can obtain a second sub-shot image by using a first camera to take a first preview image (200) when the sub-object (320) is located at a second position in the main object area (220).

[0268] In one embodiment of the present disclosure, the method of operation of the electronic device (100) may include the step (S550) of obtaining a corrected image in which the sub-object (320) is not included in the main object area (220), based on a first sub-shot image and a second sub-shot image.

[0269] In step S550, the electronic device (100) can obtain a corrected image in which the sub-object (320) is not included in the main object area (220), based on the first sub-shot image and the second sub-shot image.

[0270] Hereinafter, the operation of steps S530 to S550 will be described later in FIG. 15.

[0271] FIG. 14 is a diagram illustrating an operation to perform time correction when a second overlap end time is shorter than a reference time, in the case where a sub-object overlaps with a main object area in the current frame according to one embodiment of the present disclosure.

[0272] Referring to FIG. 1, FIG. 13 and FIG. 14, in one embodiment of the present disclosure, FIG. 14 shows a main object (210) and a sub-object (320) superimposed on a main object area (220) containing the main object (210).

[0273] In one embodiment of the present disclosure, an electronic device (100) can obtain the average speed (1420) of a sub-object (320) moving over a plurality of frames. The electronic device (100) can calculate the total length (1400) of the main object area (220) in the direction of the average speed (1420). The electronic device (100) can calculate the current length (1410) from the end of the main object area (220) in the current frame to the point where the sub-object (320) is located in the direction of the average speed (1420).

[0274] In one embodiment of the present disclosure, the frame at which the sub-object (320) exits the main object area (220) may be a second overlap termination frame. The electronic device (100) may obtain a second overlap termination time, which is the time taken until the second overlap termination frame at which the overlap between the sub-object (320) and the main object area (220) is predicted to end, by dividing the value obtained by subtracting the current length (1410) from the total length (1400) by the average speed (1420).

[0275] In one embodiment of the present disclosure, the electronic device (100) may compare a second overlap end time with a reference time and, as it is identified that the second overlap end time is shorter than the reference time, provide time information including information about the second overlap end time to the user.

[0276] In one embodiment of the present disclosure, when an electronic device (100) is configured to acquire a captured image by capturing a first preview image (200) through a first camera (110) after a preset shooting waiting time, the electronic device (100) may control the first camera (110) to acquire a captured image by capturing the first preview image (200) after a time equal to the shooting waiting time plus a second overlap end time.

[0277] Through this, the electronic device (100) can prevent the user from being provided with a degraded image of the main object (210) to be photographed, which is obscured by an unwanted sub-object (320) or a shadow (330) caused by the sub-object (320).

[0278] Additionally, in one embodiment of the present disclosure, the electronic device (100) may obtain a user's selection signal for setting the time to capture the first preview image (200) after providing the user with time information including information about the second overlap end time.

[0279] In one embodiment of the present disclosure, if the selection signal includes information in which the user selects to take a first preview image (200) after an initially set shooting waiting time, the electronic device (100) may control the first camera (110) to take a first preview image (200) during the shooting waiting time to acquire a shooting image. Additionally, the electronic device (100) may control the first camera (110) to take a first preview image (200) again after a second overlap end time to acquire a shooting image.

[0280] Through this, the electronic device (100) may provide a shooting image taken during a shooting waiting time set by the user and a shooting image in which the main object (210) is not obscured by an unwanted sub-object (320) or a shadow (330) caused by a sub-object (320).

[0281] FIG. 15 is a diagram illustrating an operation to perform frame correction according to one embodiment of the present disclosure, in which, when a sub-object overlaps with a main object area in the current frame, the second overlap end time is equal to or longer than the reference time.

[0282] Referring to FIGS. 1, 13 and 15, in one embodiment of the present disclosure, a first sub-shot image (1500), a second shot image (1510), and a corrected shot image (1520) obtained when a sub-object (320) is included at different locations within a main object area (220) are illustrated.

[0283] In one embodiment of the present disclosure, when the average speed (1530) of the sub-object (320) is slow, the second overlap end time may be identified as being equal to or longer than the reference time.

[0284] Accordingly, the electronic device (100) can acquire multiple sub-shot images in which the sub-object (320) overlaps with different areas of the main object included in the main object area (220), and acquire a corrected shot image (1520) in which the sub-object (320) is not included based on the multiple sub-shot images.

[0285] In one embodiment of the present disclosure, the electronic device (100) can obtain a first sub-shot image (1500) by using a first camera (110) to capture a first preview image (200) when a sub-object (320) is located at a first position that overlaps with a first main object (1501) within a main object area (200).

[0286] In one embodiment of the present disclosure, the electronic device (100) can obtain a second sub-shot image (1510) by taking a first preview image (200) using a first camera (110) when a sub-object (320) is located at a second position that overlaps with a second main object (1511) within a main object area (200).

[0287] In one embodiment of the present disclosure, the electronic device (100) can obtain a corrected image (1520) in which the first main object (1501) and the second main object (1511) do not overlap with the sub-object (320), based on the first sub-image (1500) and the second sub-image (1510). The electronic device (100) can obtain the corrected image (1520) through an in-painting algorithm, an image fusion algorithm, or an artificial intelligence model, and is not limited to any one of these methods.

[0288] As the electronic device (100) provides the corrected captured image (1520) to the user, the electronic device (100) can provide the user with a high-quality image in which the main object is not obscured by the sub-object (320).

[0289] Additionally, in one embodiment of the present disclosure, a sub-object (320) that overlaps with a main object area (220) may not move. In this case, the electronic device (100) may provide information to the user that the main object area (200) may be obscured by the sub-object (320) as the average speed of the sub-object (320) is identified as "0".

[0290] FIG. 16 is a drawing for explaining the operation when a user uses a head-mounted display device according to one embodiment of the present disclosure.

[0291] Referring to FIG. 1 and FIG. 16, in one embodiment of the present disclosure, FIG. 16 shows that an electronic device (1600) is implemented in the shape of a head-mounted display device.

[0292] In one embodiment of the present disclosure, the electronic device (1600) may include a camera (1610). In this case, the camera (1610) may include a first camera having a first wide angle and a second camera having a second wide angle wider than the first wide angle.

[0293] In one embodiment of the present disclosure, a user wearing an electronic device (1600) may take a specific scene using a first camera among the cameras (1610) included in the electronic device (1600). At this time, if the user is located indoors, the shadow of a sub-object not included in the first preview image of the first camera may overlap with the main object included in the image taken by the first camera due to lighting included indoors, or if the user is located outdoors, due to the sun, etc.

[0294] In one embodiment of the present disclosure, the electronic device (100) can calculate whether a sub-object or the shadow of a sub-object overlaps with a main object included in an image captured by a first camera, based on a sub-object detected by a second preview image acquired through a second camera, or calculate an overlap prediction time predicted for the overlap or an overlap end time predicted for the overlap to end.

[0295] In one embodiment of the present disclosure, the electronic device (100) may provide to the user an alarm (1620) indicating whether there is an overlap, an overlap prediction time, or an overlap end time on a display included in the electronic device (100). Additionally, the electronic device (100) may provide the above-mentioned alarm (1620) to the user as a voice or a preset vibration pattern, etc.

[0296] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood from the present disclosure by those skilled in the art to which the present disclosure pertains.

[0297] In order to solve the technical problem described above, an electronic device is provided in one embodiment of the present disclosure. The electronic device may include a first camera having a first field of view (FOV). The electronic device may include a second camera having a second field of view wider than the first field of view. The electronic device may include a memory in which a program or at least one instruction is stored. The electronic device may include at least one processor. By having at least one processor execute the program or at least one instruction stored in the memory individually or collectively, the electronic device may obtain a main object area containing a main object and a surrounding area adjacent to the main object area from a first preview image obtained through the first camera. The electronic device may detect a sub-object that is included in an area different from the main object area and the surrounding area and moves across a plurality of frames from a second preview image obtained through the second camera. The electronic device can acquire light source information regarding the position of a light source in a space containing the electronic device based on the position of a sub-object, the height of the sub-object, and the length of the shadow cast by the sub-object. The electronic device can provide time information regarding the frame in which the shadow of the sub-object, acquired based on the light source information, the position of the sub-object, and the height of the sub-object, overlaps with the main object area.

[0298] In one embodiment of the present disclosure, an electronic device may acquire a plurality of second preview images, each comprising a sub-object whose position changes over a plurality of frames, through a second camera. The electronic device may extract a sub-object and a shadow of the sub-object from each of the plurality of second preview images. The electronic device may segment the sub-object and the shadow of the sub-object from the extracted sub-object and the shadow of the sub-object. The electronic device may obtain two-dimensional shadow information for the shadow of the sub-object included in each of the plurality of second preview images by removing the segmented sub-object from the extracted sub-object and the shadow of the sub-object. Based on the two-dimensional shadow information, the electronic device may obtain the length of the shadow cast by the sub-object.

[0299] In one embodiment of the present disclosure, an electronic device may obtain three-dimensional first sub-position information of a sub-object using depth of field (DOF) from a plurality of second preview images. The electronic device may obtain a plurality of top-view images by inverse mapping the plurality of second preview images. Based on the plurality of top-view images, the electronic device may obtain top-view position information including the distance traveled by the sub-object and the position of the sub-object during a plurality of frames. Based on the sub-position information and the top-view position information, the electronic device may obtain three-dimensional position information of the sub-object including the position of the sub-object and the height of the sub-object. Based on two-dimensional shadow information and the three-dimensional position information of the sub-object, the electronic device may obtain the length of the shadow cast by the sub-object from the plurality of top-view images.

[0300] In one embodiment of the present disclosure, an electronic device can obtain light source information based on three-dimensional position information of a sub-object and the length of a shadow.

[0301] In one embodiment of the present disclosure, an electronic device can identify whether a sub-object overlaps with a main object area in the current frame. As it is identified that the sub-object does not overlap with the main object area in the current frame, the electronic device may provide time information including the time taken from the current frame to the overlap start frame in which the shadow of the sub-object is predicted to begin overlapping with the main object area as the overlap prediction time.

[0302] In one embodiment of the present disclosure, an electronic device can obtain the average velocity of a sub-object moving across a plurality of frames. The electronic device can obtain the current position of the sub-object in the current frame and the overlap position of the sub-object in the overlap start frame. The electronic device can obtain a value obtained by dividing the difference between the overlap position and the current position by the average velocity as the overlap prediction time.

[0303] In one embodiment of the present disclosure, the electronic device can acquire a first captured image by capturing a first preview image through a first camera after a preset shooting waiting time has elapsed.

[0304] In one embodiment of the present disclosure, the electronic device may include in the time information the time taken from an overlap start frame, where it is predicted that the shadow of a sub-object will begin to overlap with a main object area, to a first overlap end frame, where it is predicted that the overlap of the shadow of a sub-object with a main object area will end, as a first overlap end time.

[0305] In one embodiment of the present disclosure, as it is identified that a sub-object overlaps with a main object area in the current frame, the electronic device can identify whether a second overlap end time, which is the time taken until a second overlap end frame in which the overlap between the sub-object and the main object area is predicted to end based on light source information, the position of the sub-object, and the height of the sub-object from the current frame, is shorter than a preset reference time. As it is identified that the second overlap end time is shorter than the reference time, the electronic device can acquire a second captured image by taking a first preview image using a first camera after the second overlap end time has elapsed.

[0306] In one embodiment of the present disclosure, as the second overlap end time is identified as being equal to or longer than the reference time, the electronic device may acquire a first sub-shot image by taking a first preview image using a first camera when the sub-object is located at a first position within the main object area. The electronic device may acquire a second sub-shot image by taking a first preview image using a first camera when the sub-object is located at a second position within the main object area that is different from the first position. Based on the first sub-shot image and the second sub-shot image, the electronic device may acquire a corrected shot image in which the sub-object is not included within the main object area.

[0307] To solve the technical problem described above, one embodiment of the present disclosure provides a method for operating an electronic device. The method for operating an electronic device may include the step of acquiring a main object area containing a main object and a surrounding area adjacent to the main object area from a first preview image acquired through a first camera having a first field of view (FOV). The method for operating an electronic device may include the step of detecting a sub-object that is included in an area different from the main object area and the surrounding area and moves across a plurality of frames from a second preview image acquired through a second camera having a second field of view wider than the first field of view. The method for operating an electronic device may include the step of acquiring light source information regarding the location of a light source in a space containing the electronic device based on the position of the sub-object, the height of the sub-object, and the length of the shadow cast by the sub-object. The method for operating an electronic device may include the step of providing time information regarding a frame in which the shadow of the sub-object overlaps with the main object area, acquired based on the light source information, the position of the sub-object, and the height of the sub-object.

[0308] In one embodiment of the present disclosure, the step of detecting a sub-object may include the step of acquiring a plurality of second preview images, each containing a sub-object whose position changes over a plurality of frames through a second camera. The step of acquiring light source information may include the step of extracting a sub-object and a shadow of a sub-object from each of the plurality of second preview images. The step of acquiring light source information may include the step of segmenting the sub-object and the shadow of the sub-object from the extracted sub-object and the shadow of the sub-object. The step of acquiring light source information may include the step of removing the segmented sub-object from the extracted sub-object and the shadow of the sub-object to acquire two-dimensional shadow information for the shadow of the sub-object included in each of the plurality of second preview images. The step of acquiring light source information may include the step of acquiring the length of the shadow caused by the sub-object based on the two-dimensional shadow information.

[0309] In one embodiment of the present disclosure, the step of acquiring light source information may include acquiring three-dimensional sub-position information of a sub-object using depth of field (DOF) from a plurality of second preview images. The step of acquiring light source information may include acquiring a plurality of top-view images by inverse mapping the plurality of second preview images. The step of acquiring light source information may include acquiring top-view position information including the distance traveled by the sub-object and the position of the sub-object during a plurality of frames based on the plurality of top-view images. The step of acquiring light source information may include acquiring three-dimensional position information of the sub-object including the position of the sub-object and the height of the sub-object based on the sub-position information and the top-view position information. The step of acquiring light source information may include acquiring the length of the shadow cast by the sub-object from a plurality of top-view images based on two-dimensional shadow information and the three-dimensional position information of the sub-object. The step of acquiring light source information may include acquiring light source information based on the three-dimensional position information of the sub-object and the length of the shadow.

[0310] In one embodiment of the present disclosure, a method of operating an electronic device may include a step of identifying whether the shadow of a sub-object overlaps with a main object area in a current frame. The step of providing time information may include, as it is identified that the shadow of the sub-object does not overlap with the main object area in the current frame, the time taken from the current frame to the overlap start frame in which the shadow of the sub-object is predicted to begin overlapping with the main object area may be included in the time information as an overlap prediction time.

[0311] In one embodiment of the present disclosure, the step of providing time information may include the step of obtaining the average velocity of a sub-object moving across a plurality of frames. The step of providing time information may include the step of obtaining the current position of the sub-object in the current frame and the overlap position of the sub-object in the overlap start frame. The step of providing time information may include the step of obtaining a value obtained by dividing the difference between the overlap position and the current position by the average velocity as the overlap prediction time.

[0312] In one embodiment of the present disclosure, the method of operating an electronic device may include the step of acquiring a first captured image by capturing a first preview image through a first camera after a preset shooting waiting time has elapsed.

[0313] In one embodiment of the present disclosure, the step of providing time information may include the step of providing the time information by including the time taken from an overlap start frame, where it is predicted that the shadow of a sub-object will begin to overlap with the main object area, to an overlap end frame, where it is predicted that the overlap of the shadow of a sub-object with the main object area will end, as a first overlap end time.

[0314] In one embodiment of the present disclosure, a method of operation of an electronic device may include a step of identifying whether the shadow of a sub-object overlaps with a main object area in a current frame. As it is identified that the shadow of a sub-object overlaps with a main object area in a current frame, the method of operation of the electronic device may include a step of identifying whether a second overlap end time, which is the time elapsed from the current frame to an overlap end frame where the overlap of the shadow of the sub-object with the main object area is predicted to end, is shorter than a preset reference time. As it is identified that the second overlap end time is shorter than the reference time, the method of operation of the electronic device may include a step of acquiring a second captured image by taking a first preview image using a first camera after the second overlap end time has elapsed.

[0315] In one embodiment of the present disclosure, the method of operation of an electronic device may include the step of acquiring a first sub-shot image by taking a first preview image using a first camera when the shadow of a sub-object is located at a first position within a main object area, as identified as the second overlap end time being equal to or longer than a reference time. The method of operation of the electronic device may include the step of acquiring a second sub-shot image by taking a first preview image using a first camera when the shadow of a sub-object is located at a second position within a main object area that is different from the first position. The method of operation of the electronic device may include the step of acquiring a corrected shot image in which the sub-object is not included within the main object area, based on the first sub-shot image and the second sub-shot image.

[0316] In order to solve the aforementioned technical problem, a computer-readable recording medium may be provided on which a program for performing at least one method of an embodiment of the method of operating an electronic device disclosed in the present disclosure is recorded on a computer.

[0317] A program executed by an electronic device described in this disclosure may be implemented by hardware components, software components, and / or a combination of hardware components and software components. The program may be executed by any system capable of executing computer-readable instructions.

[0318] Software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively.

[0319] Software can be implemented as a computer program containing instructions stored on a computer-readable storage medium. Examples of computer-readable recording media include magnetic storage media (e.g., ROM (read-only memory), RAM (random-access memory), floppy disks, hard disks, etc.) and optical reading media (e.g., CD-ROMs, DVDs (Digital Versatile Discs)). Computer-readable recording media can be distributed across networked computer systems, allowing computer-readable code to be stored and executed in a distributed manner. The recording medium is readable by a computer, stored in memory, and can be executed by a processor.

[0320] Computer-readable storage media may be provided in the form of non-transitory storage media. Here, 'non-transitory storage media' simply means that it is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily. For example, 'non-transitory storage media' may include a buffer in which data is stored temporarily.

[0321] In addition, the program according to the embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product.

[0322] A computer program product may include a software program and a computer-readable storage medium on which the software program is stored. For example, a computer program product may include a product in the form of a software program (e.g., a downloadable application) that is distributed electronically through a manufacturer of an electronic device or an electronic market (e.g., Samsung Galaxy Store). For electronic distribution, at least a portion of the software program may be stored on a storage medium or temporarily created. In this case, the storage medium may be a server of the manufacturer of the electronic device, a server of the electronic market, or a storage medium of a relay server that temporarily stores the software program.

[0323] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, appropriate results can be achieved even if the described techniques are performed in a different order than described, and / or components such as the described computer system or module are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

Claims

1. In an electronic device (100), A first camera (110) having a first field of view (FOV); A second camera (120) having a second angle of view wider than the first angle of view; Memory (140) where a program or at least one instruction is stored; and It includes at least one processor (150) including a processing circuitry, By having the above at least one processor (150) execute the above program or the above at least one instruction stored in the memory (140) individually or collectively, the electronic device (100) From the first preview image obtained through the first camera (110), a main object area including a main object and a surrounding area adjacent to the main object area are obtained, and From the second preview image obtained through the second camera (120), a sub-object that is included in an area different from the main object area and the surrounding area and moves across a plurality of frames is detected, and Based on the position of the sub-object, the height of the sub-object, and the length of the shadow cast by the sub-object, light source information regarding the position of a light source in a space containing the electronic device (100) is obtained, and An electronic device (100) that provides time information for a frame in which the shadow of the sub-object overlaps with the main object area, obtained based on the light source information, the position of the sub-object, and the height of the sub-object.

2. In Paragraph 1, The above electronic device (100) is, A plurality of second preview images each including the sub-object whose position is changed over the plurality of frames are obtained through the second camera (120), and Extract the sub-object and the shadow of the sub-object from each of the plurality of second preview images above, and Segment the sub-object and the shadow of the sub-object from the extracted sub-object and the shadow of the sub-object, respectively, and By removing the segmented sub-object from the extracted sub-object and the shadow of the sub-object, two-dimensional shadow information for the shadow of the sub-object included in each of the plurality of second preview images is obtained, An electronic device (100) that obtains the length of the shadow by the sub-object based on the above two-dimensional shadow information.

3. In Paragraph 2, The above electronic device (100) is, From the plurality of second preview images above, three-dimensional sub-position information of the sub-object is obtained using the depth of field (DOF), and A plurality of top-view images are obtained by inverse mapping the plurality of second preview images above, and Based on the plurality of above-mentioned top views, top-side position information including the movement distance of the sub-object and the position of the sub-object during the plurality of frames is obtained, and Based on the above sub-position information and the above top surface position information, three-dimensional position information of the sub-object including the above position of the sub-object and the above height of the sub-object is obtained, and An electronic device (100) that obtains the length of the shadow caused by the sub-object from the plurality of top views based on the above 2D shadow information and the above 3D position information of the sub-object.

4. In Paragraph 3, The above electronic device (100) is, An electronic device (100) that acquires light source information based on the three-dimensional position information of the sub-object and the length of the shadow.

5. In any one of paragraphs 2 through 4, The above electronic device (100) is, Identify whether the above sub-object overlaps with the above main object area in the current frame, and An electronic device (100) that includes the time information as the overlap prediction time, wherein, as it is identified that the sub-object does not overlap with the main object area in the current frame, the time taken from the current frame to the overlap start frame in which the shadow of the sub-object is predicted to start overlapping with the main object area.

6. In Paragraph 5, The above electronic device (100) is, Obtaining the average speed of the sub-object moving across the plurality of frames, and Obtain the current position of the sub-object in the current frame and the nested position of the sub-object in the nested start frame, and An electronic device (100) that obtains a value obtained by dividing the difference between the above-mentioned overlap location and the above-mentioned current location by the above-mentioned average speed as the above-mentioned overlap prediction time.

7. In any one of paragraphs 1 through 6, The above electronic device (100) is, An electronic device (100) that acquires a first captured image by capturing a first preview image through the first camera (110) after the expiration of a preset shooting waiting time.

8. In any one of paragraphs 1 through 7, The above electronic device (100) is, An electronic device (100) that provides the time information including the time taken from the overlap start frame, where it is predicted that the shadow of the sub-object will begin to overlap with the main object area, to the first overlap end frame, where it is predicted that the overlap of the shadow of the sub-object with the main object area will end, as the first overlap end time.

9. In any one of paragraphs 5 through 8, The above electronic device (100) is, As it is identified that the sub-object overlaps with the main object area in the current frame, the second overlap end time, which is the time taken from the current frame to the second overlap end frame where the overlap of the sub-object with the main object area is predicted to end based on the light source information, the position of the sub-object, and the height of the sub-object, is identified as being shorter than a preset reference time. As the above second overlap end time is identified as being shorter than the above reference time, An electronic device (100) that acquires a second captured image by taking a first preview image using the first camera (110) after the second overlap end time has passed.

10. In Paragraph 9, The above electronic device (100) is, As the above second overlap end time is identified as being equal to or longer than the above reference time, When the above sub-object is located at a first position among the main object areas, the first preview image is captured using the first camera (110) to obtain a first sub-shot image, and When the above sub-object is located at a second position different from the first position in the main object area, the first camera (110) is used to capture the first preview image and obtain a second sub-shot image. An electronic device (100) that acquires a corrected image in which the sub-object is not included in the main object area, based on the first sub-shot image and the second sub-shot image.

11. In the method of operating the electronic device (100), A step (S100) of obtaining a main object area including a main object and a surrounding area adjacent to the main object area from a first preview image obtained through a first camera having a first field of view (FOV); A step (S200) of detecting a sub-object that is included in an area different from the main object area and the surrounding area and moves across a plurality of frames from a second preview image obtained through a second camera having a second angle of view wider than the first angle of view; A step (S300) of obtaining light source information regarding the position of a light source in a space containing the electronic device based on the position of the sub-object, the height of the sub-object, and the length of the shadow cast by the sub-object; and A method of operation of an electronic device comprising the step (S400) of providing time information for a frame in which the shadow of the sub-object overlaps with the main object area, obtained based on the light source information, the position of the sub-object, and the height of the sub-object.

12. In Paragraph 11, The step of detecting the above sub-object (S200) is, The method further includes the step of acquiring a plurality of second preview images, each comprising a sub-object whose position changes across the plurality of frames through the second camera. The step of acquiring the above light source information (S300) is, A step of extracting the sub-object and the shadow of the sub-object from each of the plurality of second preview images; A step of segmenting the sub-object and the shadow of the sub-object from the extracted sub-object and the shadow of the sub-object, respectively; A step of removing the segmented sub-object from the extracted sub-object and the shadow of the sub-object to obtain two-dimensional shadow information for the shadow of the sub-object included in each of the plurality of second preview images; and A method of operation of an electronic device (100) comprising the step of obtaining the length of the shadow by the sub-object based on the above two-dimensional shadow information.

13. In Paragraph 12, The step of acquiring the above light source information (S300) is, A step of obtaining three-dimensional sub-position information of the sub-object using depth of field (DOF) from the plurality of second preview images; A step of obtaining a plurality of top-view images by inverse mapping the plurality of second preview images above; A step of obtaining top surface position information including the movement distance of the sub-object and the position of the sub-object during the plurality of frames based on the plurality of top surface drawings; A step of obtaining three-dimensional position information of a sub-object, including the position of the sub-object and the height of the sub-object, based on the sub-position information and the top surface position information; Based on the above two-dimensional shadow information and the above three-dimensional position information of the sub-object, a step of obtaining the length of the shadow caused by the sub-object from the plurality of top views; and A method of operation of an electronic device (100) further comprising the step of acquiring light source information based on the three-dimensional position information of the sub-object and the length of the shadow.

14. In either Paragraph 12 or Paragraph 13, The method of operation of the above electronic device (100) is, The method further includes a step of identifying whether the sub-object overlaps with the main object area in the current frame, and The step (S400) of providing the above time information is, A method of operation of an electronic device (100) comprising the step of providing the time information including the time taken from the current frame to the overlap start frame, where the shadow of the sub-object is predicted to start overlapping with the main object area, as the overlap prediction time, as it is identified that the sub-object does not overlap with the main object area in the current frame.

15. A computer-readable recording medium having a program recorded thereon for performing the method of operation described in any one of claims 11 through 14 on a computer.

Images