Head-mounted electronic device, method, and non-transitory computer-readable storage medium for generating depth map

Abstract

This head-mounted electronic device comprises at least one processor including processing circuitry, one or more cameras having a first FOV, one or more sensors having a second FOV narrower than the first FOV, and a memory that stores one or more programs configured to be individually or collectively executed by the at least one processor and includes one or more storage media. The one or more programs may include instructions that cause the head-mounted electronic device to: identify whether an object, which corresponds to a representation in a first frame image of the first FOV acquired via the one or more cameras and is located in the second FOV, corresponds to a reference object; store depth values acquired for the object in order to generate, on the basis of the object different from the reference object, a depth map for a second frame image to be acquired via the one or more cameras; and refrain from storing the depth values on the basis of the object corresponding to the reference object.

Description

Head-worn electronic device, method, and non-transient computer-readable storage medium for generating depth maps

[0001] The present disclosure relates to a head-worn electronic device for generating a depth map, a method, and a non-transient computer-readable storage medium.

[0002] To provide an enhanced user experience, electronic devices are being developed that provide augmented reality (AR) services by displaying computer-generated information in conjunction with external objects within the real world. The electronic device may be a head-mounted electronic device that can be worn by a user. For example, the electronic device may be AR glasses and / or a head-mounted device (HMD).

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0004] A head-worn electronic device is described. The head-worn electronic device may include at least one processor comprising a processing circuit, one or more cameras having a first field of view (FOV), one or more sensors having a second field of view narrower than the first FOV, and a memory comprising one or more storage media for storing one or more programs configured to be executed individually or collectively by the at least one processor. The one or more programs may include instructions that cause the head-worn electronic device to acquire a first frame image of the first FOV of the one or more cameras through the one or more cameras. The one or more programs may include instructions that cause the head-worn electronic device to identify an object located within the second FOV of the one or more sensors that corresponds to a representation in the first frame image. The one or more programs may include instructions that cause the head-worn electronic device to identify whether the object corresponds to a reference object. The one or more programs may include instructions that cause a head-worn electronic device to store depth values ​​acquired through the one or more sensors with respect to the object while acquiring the first frame image to generate a depth map for a second frame image to be acquired through the one or more cameras, based on identifying that the object is different from the reference object. The one or more programs may include instructions that cause a head-worn electronic device to refrain from storing the depth values ​​by discarding the depth values, based on identifying that the object corresponds to the reference object.

[0005] A method is described. The method may be performed within a head-worn electronic device comprising one or more cameras having a first field of view (FOV) and one or more sensors having a second FOV narrower than the first FOV. The method may include the operation of acquiring a first frame image of the first FOV of the one or more cameras through the one or more cameras. The method may include the operation of identifying an object located within the second FOV of the one or more sensors that corresponds to a representation in the first frame image. The method may include the operation of identifying whether the object corresponds to a reference object. The method may include the operation of storing depth values ​​acquired through the one or more sensors with respect to the object while acquiring the first frame image, in order to generate a depth map for a second frame image to be acquired through the one or more cameras, based on identifying that the object is different from the reference object. The above method may include an operation of refraining from storing depth values ​​by discarding the depth values ​​based on identifying that the object corresponds to the reference object.

[0006] A non-transient computer-readable storage medium is described. The non-transient computer-readable storage medium may store one or more programs. The one or more programs may include instructions that cause the head-wearing electronic device to acquire a first frame image of the first FOV of the one or more cameras through the one or more cameras when executed by the head-wearing electronic device comprising one or more cameras having a first FOV (field of view) and one or more sensors having a second FOV narrower than the first FOV. The one or more programs may include instructions that cause the head-wearing electronic device to identify an object located within the second FOV of the one or more sensors that corresponds to a representation in the first frame image when executed by the head-wearing electronic device. The one or more programs may include instructions that cause the head-wearing electronic device to identify whether the object corresponds to a reference object when executed by the head-wearing electronic device. The above one or more programs may include instructions that cause the head-wearing electronic device to store depth values ​​acquired through the one or more sensors with respect to the object while acquiring the first frame image to generate a depth map for a second frame image to be acquired through the one or more cameras, based on identifying that the object is different from the reference object when executed by the head-wearing electronic device. The above one or more programs may include instructions that cause the head-wearing electronic device to refrain from storing the depth values ​​by discarding the depth values, based on identifying that the object corresponds to the reference object when executed by the head-wearing electronic device.

[0007] A head-wearing electronic device is described. The head-wearing electronic device may include at least one processor comprising a processing circuit, one or more cameras having a first field of view (FOV), one or more sensors having a second field of view narrower than the first FOV, and a memory comprising one or more storage media for storing one or more programs configured to be executed individually or collectively by the at least one processor. The one or more programs may include instructions that cause the head-wearing electronic device to acquire a first frame image of the first FOV of the one or more cameras through the one or more cameras. The one or more programs may include instructions that cause the head-wearing electronic device to identify an object located within the second FOV of the one or more sensors corresponding to a representation in the first frame image. The above one or more programs may include instructions that cause a head-worn electronic device to identify whether the object is being moved by comparing the first frame image with at least one second frame image prior to the first frame image. The above one or more programs may include instructions that cause a head-worn electronic device to store depth values ​​acquired through the one or more sensors with respect to the object during the acquisition of the first frame image in order to generate a depth map for a third frame image to be acquired through the one or more cameras, based on identifying that the object is not being moved. The above one or more programs may include instructions that cause a head-worn electronic device to refrain from storing the depth values ​​by discarding the depth values, based on identifying that the object is moved.

[0008] A method is described. The method may be performed within a head-worn electronic device comprising one or more cameras having a first field of view (FOV) and one or more sensors having a second FOV narrower than the first FOV. The method may include the operation of acquiring a first frame image of the first FOV of the one or more cameras through the one or more cameras. The method may include the operation of identifying an object located within the second FOV of the one or more sensors that corresponds to a representation within the first frame image. The method may include the operation of identifying whether the object is being moved by comparing the first frame image with at least one second frame image prior to the first frame image. The method may include the operation of storing depth values ​​acquired through the one or more sensors with respect to the object while acquiring the first frame image, in order to generate a depth map for a third frame image to be acquired through the one or more cameras, based on identifying that the object is not being moved. The above method may include an operation of refraining from storing depth values ​​by discarding the depth values ​​based on identifying that the object has been moved.

[0009] A non-transient computer-readable storage medium is described. The non-transient computer-readable storage medium may store one or more programs. The one or more programs may include instructions that cause the head-wearing electronic device to acquire a first frame image of the first FOV of the one or more cameras through the one or more cameras when executed by the head-wearing electronic device comprising one or more cameras having a first FOV (field of view) and one or more sensors having a second FOV narrower than the first FOV. The one or more programs may include instructions that cause the head-wearing electronic device to identify an object located within the second FOV of the one or more sensors that corresponds to a representation in the first frame image when executed by the head-wearing electronic device. The above one or more programs may include instructions that cause the head-wearing electronic device to identify whether the object is being moved by comparing the first frame image with at least one second frame image prior to the first frame image when executed by the head-wearing electronic device. The above one or more programs may include instructions that cause the head-wearing electronic device to store depth values ​​obtained through the one or more sensors with respect to the object while acquiring the first frame image in order to generate a depth map for a third frame image to be acquired through the one or more cameras, based on identifying that the object is not being moved when executed by the head-wearing electronic device.The above one or more programs may include instructions that cause the head-wearing electronic device to refrain from storing the depth values ​​by discarding the depth values ​​based on identifying that the object has moved when executed by the head-wearing electronic device.

[0010] A head-wearing electronic device is described. The head-wearing electronic device may include at least one processor comprising a processing circuit, one or more cameras having a first field of view (FOV), one or more sensors having a second field of view narrower than the first FOV, and a memory comprising one or more storage media for storing one or more programs configured to be executed individually or collectively by the at least one processor. The one or more programs may include instructions that cause the head-wearing electronic device to acquire a first frame image of the first FOV of the one or more cameras, which includes a representation of an object, through the one or more cameras. The object may be located within the first FOV of the one or more cameras and the second FOV of the one or more sensors while acquiring the first frame image. The one or more programs may include instructions that cause a head-worn electronic device to acquire depth values ​​of the object through the one or more sensors while acquiring the first frame image, and to store the depth values. The one or more programs may include instructions that cause a head-worn electronic device to acquire a second frame image of the first FOV of the one or more cameras, including a different representation of the object, through the one or more cameras after acquiring the first frame image. The object may be located within the first FOV of the one or more cameras and outside the second FOV of the one or more sensors while acquiring the second frame image.The above one or more programs may include instructions that cause a head-worn electronic device to generate a depth map for the second frame image, which provides depth information for the object located outside the second FOV of the one or more sensors while acquiring the second frame image, using the depth values.

[0011] A method is described. The method may be performed within a head-worn electronic device comprising one or more cameras having a first field of view (FOV) and one or more sensors having a second field of view narrower than the first field of view. The method may include the operation of acquiring a first frame image of the first field of view of the one or more cameras through the one or more cameras. The method may include the operation of acquiring a first frame image of the first field of view of the one or more cameras, including a representation of an object, through the one or more cameras. The object may be located within the first field of view of the one or more cameras and the second field of view of the one or more sensors while acquiring the first frame image. The method may include the operation of acquiring depth values ​​of the object through the one or more sensors while acquiring the first frame image, and storing the depth values. The above method may include the operation of acquiring a second frame image of the first FOV of the one or more cameras, including a different representation of the object, through the one or more cameras after acquiring the first frame image. The object may be located within the first FOV of the one or more cameras and outside the second FOV of the one or more sensors while acquiring the second frame image. The above method may include the operation of generating a depth map for the second frame image that provides depth information for the object located outside the second FOV of the one or more sensors while acquiring the second frame image, using the depth values.

[0012] A non-transient computer-readable storage medium is described. The non-transient computer-readable storage medium may store one or more programs. The one or more programs may include instructions that cause the head-wearing electronic device to acquire a first frame image of the first FOV of the one or more cameras, which includes a representation of an object, through the one or more cameras, when executed by the head-wearing electronic device comprising one or more cameras having a first FOV (field of view) and one or more sensors having a second FOV narrower than the first FOV. The object may be located within the first FOV of the one or more cameras and the second FOV of the one or more sensors while acquiring the first frame image. The one or more programs may include instructions that cause the head-wearing electronic device to acquire depth values ​​of the object through the one or more sensors and store said depth values ​​while acquiring the first frame image, when executed by the head-wearing electronic device. The above one or more programs may include instructions that cause the head-wearing electronic device, when executed by the head-wearing electronic device, to acquire the first frame image and then, through the one or more cameras, acquire a second frame image of the first FOV of the one or more cameras containing a different representation of the object. The object may be located within the first FOV of the one or more cameras and outside the second FOV of the one or more sensors while acquiring the second frame image.The above one or more programs may include instructions that cause the head-wearing electronic device to generate a depth map for the second frame image, which provides depth information for the object located outside the second FOV of the one or more sensors while acquiring the second frame image, using the depth values ​​when executed by the head-wearing electronic device.

[0013] Figure 1 illustrates an example of an image and depth values ​​obtained to generate a depth map.

[0014] Figure 2 is a simplified block diagram of an exemplary head-worn electronic device.

[0015] FIG. 3 is a flowchart illustrating exemplary operations of a head-worn electronic device for identifying an object corresponding to a visual object within a first frame image.

[0016] FIG. 4 illustrates an example of a first frame image and first depth values.

[0017] FIG. 5 is a flowchart illustrating exemplary operations of a head-worn electronic device for identifying whether an object corresponds to a reference object.

[0018] Figure 6 illustrates an example of discarding depth values ​​of an object corresponding to a reference object.

[0019] Figure 7 illustrates an example of discarding depth data of a distorted area within a visual object corresponding to an object.

[0020] FIG. 8 is a flowchart illustrating exemplary operations of a head-worn electronic device for identifying the reliability values ​​of each of the depth values ​​of an object corresponding to a reference object.

[0021] FIG. 9 is a flowchart illustrating exemplary operations of a head-worn electronic device for identifying whether an object has been moved.

[0022] Figure 10 illustrates an example of discarding depth values ​​of a moved object.

[0023] FIG. 11 is a flowchart illustrating exemplary operations of a head-worn electronic device for generating a depth map.

[0024] FIG. 12 illustrates an example of generating a depth map for a second frame image using depth values ​​obtained based on a first frame image.

[0025] FIG. 13 is a block diagram of an electronic device in a network environment according to various embodiments.

[0026] FIG. 14a shows an example of a perspective view of a wearable device.

[0027] FIG. 14b shows an example of one or more hardware components placed within a wearable device.

[0028] FIGS. 15a to 15b show an example of the appearance of a wearable device.

[0029] Figure 16 shows an example of a block diagram of a wearable device.

[0030] Figure 17 shows an example of a block diagram of an electronic device for displaying an image in virtual space.

[0031] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0032] Figure 1 illustrates an example of an image and depth values ​​obtained to generate a depth map.

[0033] Referring to FIG. 1, the head-wearing electronic device (100) may include a head-mounted display (HMD) that can be worn on the head of a user (110). The head-wearing electronic device (100) may be described as a head-mounted display (HMD) device, a headgear electronic device, a glasses-type (or goggle-type) electronic device, a video see-through (VST) or visible see-through (VST) device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device.

[0034] A head-worn electronic device (100) may include one or more cameras (e.g., one or more cameras (230) of FIG. 2) and one or more sensors (e.g., one or more sensors (240) of FIG. 2). One or more cameras may have a first field of view (FOV). For example, one or more sensors may be described as depth sensors (e.g., indirect time of flight (I-TOF) and / or direct time of flight (D-TOF)). One or more sensors may have a second field of view that is narrower than the first field of view. Each of the one or more sensors may have a different field of view. Each of the one or more sensors may have a different precision.

[0035] A head-worn electronic device (100) can acquire an image (115) of a first FOV of one or more cameras of the space in front of a user (110) through one or more cameras. While acquiring the image (115), the head-worn electronic device (100) can acquire depth values ​​of an area (120) of a second FOV within the image (115) of the first FOV through one or more sensors. As the second FOV is narrower than the first FOV, the area (120) of the second FOV may be smaller than the image (115) of the first FOV. As the area (120) of the second FOV is smaller than the image (115) of the first FOV, the head-wearing electronic device (100) may not be able to obtain depth values ​​for the remaining area excluding the area (120) of the second FOV within the image (115) of the first FOV through one or more sensors. As the head-wearing electronic device (100) may not obtain depth values ​​for the remaining area excluding the area (120) of the second FOV within the image (115) of the first FOV through one or more sensors, generating a depth map for the image (115) may have errors.

[0036] One or more sensors may have measuring spots (125). Since one or more sensors have a second FOV, the measuring spots (125) of one or more sensors may be distributed within the area (120) of the second FOV in the image (115). Since each of the one or more sensors has a different FOV and / or precision, the measuring spots (125) may overlap each other. For example, the measuring spots (125) may be distributed within an area different from the area (120) of the second FOV. The measuring spots (125) may have a density lower than the density of the pixels of the image (115). As the measurement points (125) have a density lower than the density of pixels in the image (115), the head-wearing electronic device (100) may not be able to obtain depth values ​​for the area excluding the measurement points (125) (or the area between the measurement points (125)) within the area (120) of the second FOV. For example, the head-wearing electronic device (100) may not be able to obtain depth values ​​of an external object corresponding to a representation contained within the area excluding the measurement points (125) (or the area between the measurement points (125)) within the area (120) of the second FOV. Because the number of measurement points (125) is limited, the depth values ​​of the external object may be inaccurate or unreliable depending on the size of the external object. Since the head-worn electronic device (100) fails to obtain depth values ​​for areas excluding measurement points (125) within the area (120) of the second FOV, generating a depth map for the image (115) may have errors.

[0037] A method to resolve these errors related to generating a depth map for an image (115) may be required. To resolve these errors, a head-worn electronic device (100) may use depth values ​​obtained through one or more sensors while acquiring a previous (or immediately preceding) frame image of the image (115). To use the depth values ​​obtained through one or more sensors while acquiring a previous (or immediately preceding) frame image of the image (115), the head-worn electronic device (100) may identify an object corresponding to a representation within the previous (or immediately preceding) frame image of the image (115).

[0038] A head-worn electronic device (100) may perform operations that are exemplified within the description of FIGS. 3 through 12 to generate a depth map. The head-worn electronic device (100) may include components for performing said operations. said components may be exemplified within the description of FIG. 2.

[0039] Figure 2 is a simplified block diagram of an exemplary head-worn electronic device.

[0040] Referring to FIG. 2, the head-wearing electronic device (200) may be described as a head-mount display (HMD) device, headgear electronic device, glasses-type (or goggle-type) electronic device, video see-through (VST) device, extended reality (XR) device, virtual reality (VR) device, and / or augmented reality (AR) device. An example of the structure of the head-wearing electronic device (200) is described with reference to FIG. 14a, FIG. 14b, FIG. 15a and / or FIG. 15b. The head-wearing electronic device (200) may include at least a part of the electronic device (1301) of FIG. 13 or correspond to at least a part of the electronic device (1301) of FIG. 13. The head-wearing electronic device (200) may include at least one processor (210), a memory (220), one or more cameras (230), and one or more sensors (240).

[0041] According to one embodiment, at least one processor (210) may include a processing circuit. At least one processor (210) may include a central processing unit (e.g., including a processing circuit). At least one processor (210) may include a graphic processing unit (e.g., including a processing circuit) and a neural processing unit (e.g., including a processing circuit). At least one processor (210) may include at least a part of the processor (1320) of FIG. 13 or correspond to at least a part of the processor (1320) of FIG. 13. For example, at least one processor (210) may be configured to control a memory (220), one or more cameras (230), and one or more sensors (240). At least one processor (210) may be configured to execute instructions stored in memory (220) individually or collectively to cause the head-wearing electronic device (200) (or head-wearing electronic device (100)) to perform at least some of the operations illustrated in the description of FIG. 1. At least one processor (210) may be configured to execute instructions stored in memory (220) individually or collectively to cause the head-wearing electronic device (200) to perform at least some of the operations illustrated in the descriptions of FIG. 3 through 13.

[0042] According to one embodiment, the memory (220) may include one or more storage media. The memory (220) may store various data used by at least one component of the head-wearing electronic device (200) (e.g., at least one processor (210), memory (220), one or more cameras (230), and / or one or more sensors (240)). For example, the memory (220) may include at least a portion of the memory (1330) of FIG. 13 or correspond to at least a portion of the memory (1330) of FIG. 13. For example, the data may include input data or output data for software and related commands. The memory (220) may include volatile memory or non-volatile memory.

[0043] According to one embodiment, one or more cameras (230) may include one or more light sensors (e.g., a charged coupled device (CCD) sensor and / or a complementary metal oxide semiconductor (CMOS) sensor) that generate an electrical signal indicating the color and / or brightness of light. For example, one or more cameras (230) may be described as image sensors. For example, one or more cameras (230) may include at least a part of the camera module (1380) of FIG. 13 or correspond to at least a part of the camera module (1380) of FIG. 13. For example, one or more cameras (230) may be available to acquire images of the space (or surrounding environment) in front of the head-worn electronic device (200). For example, one or more cameras (230) may have a first FOV. For example, at least a part of one or more cameras (230) may have a FOV corresponding to the field of view (FOV) of the user's eye. For example, the FOV of some of the one or more cameras (230) may be different from the FOV of other parts of the one or more cameras (230).

[0044] According to one embodiment, one or more sensors (240) may be configured to acquire depth values ​​of an object corresponding to a visual object in an image acquired through one or more cameras (230). One or more sensors (240) may include light-emitting elements and light-receiving elements. For example, one or more sensors (240) may include at least a part of the sensor module (1376) of FIG. 13 or correspond to at least a part of the sensor module (1376) of FIG. 13. For example, one or more sensors (240) may acquire depth information of the object by using the difference between the light emitted by the light-emitting elements and the light received by the light-receiving elements. For example, one or more sensors (240) may include an I-TOF (indirect time of flight) sensor and / or a D-TOF (direct time of flight) sensor. However, they are not limited thereto. For example, one or more sensors (240) may be operably or operatively coupled with at least one processor (210). For example, each of the one or more sensors (240) may have a different FOV. For example, each of the one or more sensors (240) may have a different precision.

[0045] According to one embodiment, the head-wearing electronic device (200) illustrated in the description of FIG. 2 may perform at least some of the operations illustrated in the descriptions of FIG. 3 through 12. The operations illustrated in the descriptions of FIG. 3 through 12 may be caused by (or within) the head-wearing electronic device (200) under the control of at least one processor (210).

[0046] FIG. 3 is a flowchart illustrating exemplary operations of a head-worn electronic device for identifying an object corresponding to a visual object within a first frame image.

[0047] Referring to FIG. 3, in operation 300, according to one embodiment, at least one processor (210) can acquire a first frame image of a first FOV of one or more cameras (230) (e.g., the first frame image (400) of FIG. 4) through one or more cameras (230). For example, the first frame image can be described as an image of the space in front of the head-worn electronic device (200).

[0048] According to one embodiment, at least one processor (210) can acquire first depth values ​​of an object within a second FOV through one or more sensors (240) while acquiring a first frame image. For example, since one or more sensors (240) have a second FOV that is narrower than the first FOV of one or more cameras (230), depth values ​​of an object located within the first FOV and outside the second FOV cannot be acquired.

[0049] According to one embodiment, in operation 310, at least one processor (210) can identify an object located within a second FOV of one or more sensors (240) corresponding to a visual object in a first frame image. For example, the visual object may be defined as a representation. At least one processor (210) can identify an object located within the second FOV based on the first frame image to determine whether to store (or accumulate) first depth values ​​to generate a depth map for a second frame image (e.g., the second frame image (1210) of FIG. 12) that follows (or immediately follows) the first frame image. Since at least one processor (210) cannot obtain depth values ​​of objects located outside the second FOV in the first frame image, it may not be required to obtain depth values ​​of objects located within the first FOV and outside the second FOV.

[0050] According to one embodiment, an object may be described as an external object, and a visual object within a first frame image may be described as an image representation representing the shape of the object. At least one processor (210) can identify an object located in the space in front of the head-worn electronic device (200) using the first frame image. For example, at least one processor (210) can identify at least one object corresponding to a visual object within the first frame image by performing segmentation processing of the first frame image. At least one processor (210) can perform segmentation processing using a trained model. For example, the trained model may include an artificial intelligence model, a deep learning model, and / or a machine learning model. At least one processor (210) can classify and recognize at least one object included in the first frame image by performing image analysis using the trained model.

[0051] According to one embodiment, in operation 320, at least one processor (210) can identify whether an object identified in operation 310 corresponds to a reference object. For example, the reference object may be described as a dynamic object (or a movable object, or an object likely to move, or a non-fixed object). At least one processor (210) can identify whether the identified object corresponds to the reference object by comparing the identified object with the reference object. As an example without limitation, at least one processor (210) can identify whether the identified object corresponds to the reference object by using a table that has the identified object as an input value and whether the identified object corresponds to the predetermined reference object as an output value, for a predetermined reference object. For example, the table may be referred to as a look-up table. A first frame image and first depth values ​​obtained based on the first frame image are exemplified within the description of FIG. 4.

[0052] FIG. 4 illustrates an example of a first frame image and first depth values.

[0053] Referring to FIG. 4, at least one processor (210) can acquire a first frame image (400) of a first FOV through one or more cameras (230). The first frame image (400) may include visual objects (415). Objects corresponding to the visual objects (415) may be located within the first FOV of one or more cameras (230) in the space in front of the head-worn electronic device (200).

[0054] According to one embodiment, at least one processor (210) can obtain depth values ​​of at least one measurement point (410) of one or more sensors (240) within a first frame image (400). For example, at least one measurement point (410) may be located (or distributed) within the area (405) of the second FOV within the first frame image (400) as one or more sensors (240) have a second FOV. For example, at least one measurement point (410) may overlap with at least a portion of at least one visual object (415) within the area (405) of the second FOV within the first frame image (400). At least one processor (210) can obtain depth values ​​of objects corresponding to at least one visual object (415) within the area (405) of the second FOV based on the depth values ​​of at least one measurement point (410).

[0055] According to one embodiment, some of the objects corresponding to the visual objects (415) may be located within the second FOV of the first frame image (400) obtained through one or more cameras (230). At least one processor (210) may obtain first depth values ​​of the objects corresponding to the visual objects (415-1, 415-2, 415-3, 415-4) within the area (405) of the second FOV of the first frame image (400) (e.g., first depth values ​​of at least one measurement point (410) on the visual objects (415-1, 415-2, 415-3, 415-4) within the area (405) of the second FOV).

[0056] According to another embodiment, at least one measurement point (410) may be located outside the area (405) of the second FOV by each of one or more sensors having different FOVs and / or precisions. For example, at least one processor (210) may obtain depth values ​​of objects outside the area (405) of the second FOV by some of the at least one measurement point (410) being located outside the area (405) of the second FOV.

[0057] According to one embodiment, some other objects among the objects corresponding to the visual objects (415) may be located outside the second FOV of the first frame image (400) obtained through one or more cameras (230). At least one processor (210) may not be able to obtain depth values ​​of some other objects among the objects corresponding to the visual objects (415) of the first frame image (400) that are located outside the second FOV. For example, at least one processor (210) may identify an object located within the second FOV from which the first depth values ​​can be obtained in order to determine whether to store (or accumulate) the first depth values. Whether to store (or accumulate) the first depth values ​​of the identified object depending on whether the identified object corresponds to a reference object is exemplified in the description of FIG. 5.

[0058] FIG. 5 is a flowchart illustrating exemplary operations of a head-worn electronic device for identifying whether an object corresponds to a reference object.

[0059] Referring to FIG. 5, in operation 500, at least one processor (210) can identify whether the identified object corresponds to a reference object. For example, at least one processor (210) can identify at least one object corresponding to a visual object within the first frame image by performing segmentation processing of the first frame image. At least one processor (210) can perform segmentation processing using a trained model. At least one processor (210) can classify and recognize at least one object included in the first frame image by performing image analysis using a trained model.

[0060] For example, an object corresponding to a reference object may be described as a dynamic object. The object corresponding to the reference object may not be moved or fixed within a second frame image following (or immediately following, or next frame image) the first frame image. For example, action 500 may correspond to action 320 of FIG. 3.

[0061] According to another embodiment, at least one processor (210) can identify whether the first frame image is distorted (or whether there are blurred areas). For example, the distortion may be described as distortion caused by one or more cameras (230). For example, the distortion may include barrel distortion, pincushion distortion, distortion due to blurring, and / or other distortions.

[0062] According to one embodiment, in operation 510, at least one processor (210) can identify that the object corresponds to a reference object, a distorted area within the first frame image, and / or a blurred area within the first frame image. Based on the object corresponding to the reference object (or the distorted area within the first frame image, or the blurred area within the first frame image), at least one processor (210) may refrain from (or bypass, or stop, or not store) the first depth values ​​obtained through one or more sensors (240) with respect to the object in the first frame image.

[0063] For example, an object corresponding to a reference object may be described as a dynamic object. The object corresponding to the reference object may not be moved or fixed within a second frame image that follows (or is immediately after, or is the next frame image) the first frame image. For example, since the first depth values ​​of the object corresponding to the reference object cannot be used to generate a depth map for the second frame image, the first depth values ​​of the object corresponding to the reference object may not be stored.

[0064] For example, since the depth map for the second frame image generated using the second depth values ​​of the distorted region (or blurred region) in the first frame image may have errors, it may not be required to store (or accumulate) the second depth values. For example, at least one processor (210) may refrain from storing (or accumulating) the second depth values ​​of the distorted region (or blurred region) in the first frame image.

[0065] According to one embodiment, in operation 520, at least one processor (210) can identify that the object is different from (or does not correspond to) the reference object, an undistorted area within the first frame image, and / or an unblurred area within the first frame image. At least one processor (210) can store (or accumulate) first depth values ​​obtained through one or more sensors (240) with respect to the object of the first frame image based on the object that is different from (or does not correspond to) the reference object (or an undistorted area within the first frame image, or an unblurred area within the first frame image).

[0066] For example, a reference object and other objects may be described as static objects. The reference object and other objects may not move or may be fixed within a second frame image, which is the frame following the first frame image (or the frame immediately following). For example, the first depth values ​​of an object corresponding to a reference object that is not moved within the second frame image may be used to generate a depth map for the second frame image, and thus the first depth values ​​of the object corresponding to the reference object may be stored (or accumulated). Among the depth values ​​obtained based on the first frame image, discarding the first depth values ​​of the object corresponding to the reference object and storing (or accumulating) the first depth values ​​of the reference object and other objects are exemplified in the description of FIG. 6.

[0067] Figure 6 illustrates an example of discarding depth values ​​of an object corresponding to a reference object.

[0068] Referring to FIG. 6, the first frame image (400) may include visual objects (415). At least one processor (210) may identify objects (or at least some of the objects) located within the second FOV that correspond to the visual objects (415). For example, the objects (or at least some of the objects) located within the second FOV may correspond to the visual objects (415) (or at least some of the visual objects) within the area (405) of the second FOV. For example, at least one processor (210) may identify among the objects (or at least some of the objects) within the second FOV that correspond to a reference object (e.g., an object corresponding to the visual object (415-1) and / or an object corresponding to the visual object (415-2)).

[0069] According to one embodiment, at least one processor (210) may obtain first depth values ​​of an object corresponding to a reference object based on measurement points (600) on a visual object (415-1, 415-2) of an object corresponding to a reference object (e.g., a chair and / or a curtain). For example, an object corresponding to a reference object may be described as an object that is movable, or not fixed, or has a high probability of moving. For example, since the object corresponding to a reference object has a high probability of moving in a second frame, which is the frame following the first frame, the position of the object corresponding to a reference object at the time the first frame image is acquired may differ from the position at the time the second frame image is acquired. Because the position of the object corresponding to a reference object in the first frame image may differ from the position of the object corresponding to a reference object in the second frame image, the first depth values ​​of the object corresponding to a reference object may not be used to generate a depth map for the second frame image. At least one processor (210) may not store (or accumulate) first depth values ​​of objects corresponding to reference objects that will not be used to generate a depth map for a second frame image (or first depth values ​​of measurement points (600) on visual objects (415-1, 415-2)). According to one embodiment, measurement points (600) on visual objects (415-1, 415-2) corresponding to objects corresponding to reference objects among the objects identified by performing segmentation may not be accumulated. For example, objects corresponding to visual objects (415-1, 415-2) (e.g., curtains, chairs) may be determined to be highly movable or movable objects. For example, at least one processor (210) may exclude measurement points on visual objects (415-1, 415-2) (e.g., curtains, chairs) from accumulation.

[0070] According to one embodiment, at least one processor (210) can obtain first depth values ​​of a reference object and other objects based on measurement points (605) on visual objects (415-3, 415-4) of a reference object and other objects. For example, the reference object and other objects may be described as objects that do not move, or are fixed, or have a low probability of moving. As the positions of the reference object and other objects in the first frame image correspond to the positions of the reference object and other objects in the second frame image, the first depth values ​​of the reference object and other objects may be used to generate a depth map for the second frame image. At least one processor (210) may store (or accumulate) the first depth values ​​of the reference object and other objects to be used to generate a depth map for the second frame image.

[0071] According to one embodiment, the first frame image (400) may include a distorted region on a visual object (415-3, 415-4) corresponding to a reference object and another object. For example, a depth map for a second frame image generated using first depth values ​​of the visual object (415-3, 415-4) including the distorted region may have an error. To resolve this error, identifying the distorted region on the visual object corresponding to a reference object and another object within the first frame image is exemplified in the description of FIG. 7.

[0072] Figure 7 illustrates an example of discarding depth data of a distorted area within a visual object corresponding to an object.

[0073] Referring to FIG. 7, the first frame image (400) may include a visual object (415-3, 415-4) corresponding to an object different from (or not corresponding to) the reference object. At least one processor (210) may identify whether the visual object (415-3, 415-4) includes a distorted area in order to determine whether to store (or accumulate) first depth values ​​of the object different from the reference object (e.g., first depth values ​​of measurement points (705, 710) on the visual object (415-3, 415-4). For example, at least one processor (210) may identify a distorted area (700) (or a blurred area) within the visual object (415-3). The distorted area (700) within the visual object (415-3) may have a blurred, curved, or distorted shape.

[0074] According to one embodiment, at least one processor (210) may obtain second depth values ​​of the distorted area (700) (or blurred area) within the visual object (415-3) based on measurement points (705) on the distorted area (700) (or blurred area) within the visual object (415-3). Since the second depth values ​​of the distorted area (700) (or blurred area) within the visual object (415-3) (e.g., second depth values ​​of measurement points (705) on the visual object (415-3)) may differ from the depth values ​​for the object corresponding to the visual object (415-3) due to the distortion of the first frame image (400), the second depth values ​​may not be used to generate a depth map for the second frame image. At least one processor (210) may not store (or accumulate) second depth values ​​by discarding second depth values ​​of a distorted area (700) (or blurred area) that will not be used to generate a depth map for a second frame image (e.g., second depth values ​​of measurement points (705) on a visual object (415-3).

[0075] According to one embodiment, at least one processor (210) can obtain third depth values ​​of undistorted regions within a visual object (415-3, 415-4) based on measurement points (710) on undistorted regions within a visual object (415-3, 415-4). The third depth values ​​of undistorted regions within a visual object (415-3, 415-4) (e.g., third depth values ​​of measurement points (710) on a visual object (415-3, 415-4)) correspond to depth values ​​of an object corresponding to the visual object (415-3, 415-4), so that the third depth values ​​can be used to generate a depth map for a second frame image. At least one processor (210) can store (or accumulate) third depth values ​​of undistorted regions within a visual object (415-3, 415-4) to be used to generate a depth map for a second frame image.

[0076] According to another embodiment, at least one processor (210) may further use video image, audio information, posture of head-worn electronic device (200), and / or information from one or more cameras (230) to determine depth values ​​to be used among first depth values ​​to generate a depth map for a second frame image.

[0077] According to another embodiment, the first depth values ​​of a reference object and other objects corresponding to the visual objects (415-3, 415-4) (e.g., the first depth values ​​of the measurement points (705, 710)) may have different reliability values. For example, a depth map for a second frame image generated using depth values ​​that have a relatively low reliability value among the first depth values ​​may have an error. To resolve this error, identifying the reliability values ​​of the first depth values ​​is exemplified in the description of FIG. 8.

[0078] FIG. 8 is a flowchart illustrating exemplary operations of a head-worn electronic device for identifying the reliability values ​​of each of the depth values ​​of an object corresponding to a reference object.

[0079] Referring to FIG. 8, in operation 800, at least one processor (210) can identify that an object located within the second FOV is different from the reference object (or does not correspond to the reference object). Based on identifying the object different from the reference object (or the object not corresponding to the reference object), at least one processor (210) can perform the following operations (operations 810 to 840) to determine whether to store (or accumulate) depth values ​​obtained through one or more sensors (240) with respect to the object of the first frame image.

[0080] According to one embodiment, in operation 810, at least one processor (210) can identify reliability values ​​for each of the first depth values ​​of the reference object and the other object based on identifying the reference object and the other object. For example, the reliability values ​​for each of the first depth values ​​can be described as the reliability values ​​of each of the first depth values ​​of the measurement points on the visual object corresponding to the reference object and the other object within the first frame image. For example, the reliability values ​​of the first depth values ​​can be proportional to the accuracy of the first depth values ​​with respect to the actual depth values ​​of the reference object and the other object. For example, identifying the reliability values ​​of the first depth values ​​can be defined by the analysis of the measurement points themselves. At least one processor (210) can define the accumulation range of measurement points and valid measurement points by setting conditions.

[0081] According to one embodiment, in operation 820, at least one processor (210) can compare the reliability values ​​of the first depth values ​​with a reference reliability value. The reference reliability value may be set to determine the depth values ​​among the first depth values ​​to be used to generate the second frame image. The reference reliability value may be set by a user or changed.

[0082] According to one embodiment, in operation 830, at least one processor (210) can identify second depth values ​​among the first depth values ​​that have a confidence value less than a reference confidence value. The second depth values ​​may be described as relatively inaccurate depth values ​​for an object, or may differ from the depth values ​​of the object. A depth map for a second frame image generated using second depth values ​​that are relatively inaccurate depth values ​​for an object (or different from the depth values ​​of the object) may have an error. At least one processor (210) may not use the second depth values ​​to generate a depth map for the second frame image. At least one processor (210) may not store (or accumulate) the second depth values ​​by discarding the second values ​​that will not be used to generate a depth map for the second frame image.

[0083] According to one embodiment, in operation 840, at least one processor (210) can identify third depth values ​​having a confidence value greater than or equal to a reference confidence value among the first depth values. The third depth values ​​can be described as depth values ​​relatively accurate with respect to an object, or can correspond to depth values ​​of an object. As the third depth values ​​are described as depth values ​​relatively accurate with respect to an object (or correspond to depth values ​​of an object), the third depth values ​​can be used to generate a depth map for a second frame image. At least one processor (210) can store (or accumulate) the third depth values ​​to be used to generate a depth map for a second frame image.

[0084] According to another embodiment, at least one processor (210) can identify whether an object in the first frame image has moved by comparing the first frame image with at least one third frame image prior to (or immediately prior to) the first frame image. A depth map for a second frame image generated using first depth values ​​of the moved object in the first frame image may have an error. To resolve this error, identifying whether an object in the first frame image has moved is exemplified in the description of FIG. 9.

[0085] FIG. 9 is a flowchart illustrating exemplary operations of a head-worn electronic device for identifying whether an object has been moved.

[0086] Referring to FIG. 9, in operation 900, at least one processor (210) can acquire a first frame image of a first FOV of one or more cameras (230) (e.g., the first frame image (400) of FIG. 4) through one or more cameras (230). For example, operation 900 may correspond to operation 300 of FIG. 3.

[0087] According to one embodiment, in operation 910, at least one processor (210) can identify an object located within a second FOV of one or more sensors that corresponds to a visual object in a first frame image. For example, operation 910 may correspond to operation 310 of FIG. 3.

[0088] According to one embodiment, in operation 920, at least one processor (210) can identify whether an object in the first frame image has moved by comparing the first frame image with at least one third frame image prior to (or immediately prior to) the first frame image (e.g., the third frame image (1000) of FIG. 10). In an example that is not limited to, at least one processor (210) can extract features of the object by identifying the object in the at least one third frame image. At least one processor (210) can identify whether the object has moved by detecting a region in the first frame image that is similar to the features of the object in the third frame image. According to another embodiment, at least one processor (210) can identify a moved object in the first frame image based on the motion vector of the object in the third frame image. However, it is not limited thereto.

[0089] According to one embodiment, in operation 1030, at least one processor (210) can identify that an object in the first frame image has been moved by comparing the first frame image and the third frame image. The moved object in the first frame image may be moved, or not fixed, or may have a high probability of being moved in the second frame image. Since the depth map for the second frame image generated using the first depth values ​​of the moved object in the first frame image may have errors, the first depth values ​​of the moved object in the first frame image may not be used to generate the depth map for the second frame image. As the first depth values ​​of the moved object in the first frame image are not used to generate the depth map for the second frame image, the at least one processor (210) may refrain from storing (or accumulating) the first depth values ​​of the moved object in the first frame image by discarding the first depth values ​​of the moved object in the first frame image.

[0090] According to one embodiment, in operation 940, at least one processor (210) can identify whether an object in the first frame image has not moved (or whether the object has stopped) by comparing the first frame image with the third frame image. Based on identifying whether an object in the first frame image has not moved, at least one processor (210) can store (or accumulate) first depth values ​​of the unmoved object in the first frame image. Since the first depth values ​​of the unmoved object in the first frame image can be used to generate a depth map for the second frame image, the first depth values ​​of the unmoved object in the first frame image can be stored (or accumulated) to generate a depth map for the second frame image. Discarding or storing (or accumulating) the first depth values ​​depending on whether the object in the first frame image has moved is exemplified in the description of FIG. 10.

[0091] Figure 10 illustrates an example of discarding depth values ​​of a moved object.

[0092] Referring to FIG. 10, according to one embodiment, at least one processor (210) can acquire at least one third frame image (1000) prior to (or a frame image prior to the first frame image (400)) through one or more cameras (230). The third frame image (1000) may include other visual objects (1005) corresponding to objects. At least one processor (210) can acquire a first frame image (400) following (or immediately following) the third frame image (1000) through one or more cameras (230). At least one processor (210) can identify whether objects corresponding to the visual objects (415) (or other visual objects (1005)) have moved by comparing the first frame image (400) with the third frame image (1000). At least one processor (210) can identify whether the objects have been moved by comparing each of the visual objects (415) in the first frame image (400) with each of the other visual objects (1005) in the third frame image (1000).

[0093] According to one embodiment, at least one processor (210) can identify that an object corresponding to a visual object (415-1, 415-2) in the first frame image (400) has been moved by comparing the first frame image (400) and the third frame image (1000). At least one processor (210) can obtain first depth values ​​of an object corresponding to a visual object (415-1, 415-2) based on measurement points (1015) on the moved visual object (415-1, 415-2) in the first frame image (400). Since the object corresponding to the visual object (415-1, 415-2) may not be moved or fixed within the second frame image, the first depth values ​​of the object corresponding to the visual object (415-1, 415-2) (e.g., the first depth values ​​of the measurement points (1015)) may not be used to generate a depth map for the second frame image. At least one processor (210) may not store (or accumulate) the first depth values ​​of the object corresponding to the visual object (415-1, 415-2) (e.g., the depth values ​​of the measurement points (1015)) by discarding the first depth values ​​of the object corresponding to the visual object (415-1, 415-2) that are not used to generate a depth map for the second frame image.

[0094] According to one embodiment, at least one processor (210) can identify that an object corresponding to a visual object (415-3, 415-4) in the first frame image (400) has not moved by comparing the first frame image (400) and the third frame image (1000). At least one processor (210) can obtain first depth values ​​of an object corresponding to a visual object (415-3, 415-4) based on measurement points (1010) on the visual object (415-3, 415-4) in the first frame image (400). Since the object corresponding to the visual object (415-3, 415-4) may not be fixed or moved within the second frame image, the first depth values ​​of the object corresponding to the visual object (415-3, 415-4) (e.g., the first depth values ​​of the measurement points (1010)) may be used to generate a depth map for the second frame image. At least one processor (210) may store (or accumulate) the first depth values ​​of the object corresponding to the visual object (415-3, 415-4) (e.g., the first depth values ​​of the measurement points (1010)) that are used to generate a depth map for the second frame image.

[0095] According to another embodiment, at least one processor (210) can identify whether an area within a visual object (415-3, 415-4) is distorted by comparing a first frame image (400) and a third frame image (1000) and identifying that an object corresponding to a visual object (415-3, 415-4) within the first frame image (400) has not moved. At least one processor (210) can refrain from storing (or accumulating) second depth values ​​by discarding the second depth values ​​of the distorted area within the visual object (415-3, 415-4) from the first depth values ​​of the object corresponding to the visual object (415-3, 415-4) (e.g., first depth values ​​of measurement points (1010)) based on identifying the distorted area within the visual object (415-3, 415-4). At least one processor (210) may store (or accumulate) a third depth of an undistorted area within a visual object (415-3, 415-4) among first depth values ​​of an object corresponding to the visual object (415-3, 415-4), based on identifying an undistorted area within the visual object (415-3, 415-4). For example, regarding determining which depth values ​​to store (or accumulate) (or which depth values ​​to discard) among the first depth values ​​depending on whether an area within the visual object (415-3, 415-4) is distorted, the description of FIG. 5 may be referenced.

[0096] According to another embodiment, at least one processor (210) can identify the reliability values ​​of the first depth values ​​(e.g., the first depth values ​​of the measurement points (1010)) of the object corresponding to the visual object (415-3, 415-4) in the first frame image (400) based on identifying that the object corresponding to the visual object (415-3, 415-4) in the first frame image (400) has not moved by comparing the first frame image (400) and the third frame image (1000). At least one processor (210) can refrain from storing (or accumulating) the second depth values ​​by discarding the second depth values ​​among the first depth values ​​based on identifying the second values ​​among the first depth values ​​of the object corresponding to the visual object (415-3, 415-4) that have reliability values ​​less than the reference reliability value. At least one processor (210) may store (or accumulate) third depth values ​​among first depth values ​​(e.g., first depth values ​​of measurement points (1010)) of an object corresponding to a visual object (415-3, 415-4) based on identifying third values ​​having a reliability value greater than or equal to a reference reliability value. For example, regarding determining which depth values ​​to store (or accumulate) (or which depth values ​​to discard) among the first depth values ​​based on whether the reliability value of each of the first depth values ​​is less than the reference reliability value, the description of FIG. 8 may be referenced.

[0097] According to one embodiment, at least one processor (210) may generate a depth map for a second frame image using first depth values ​​(or third depth values) of an unmoved object in a first frame image (400). Generating a depth map for a second frame image is illustrated in the description of FIG. 11.

[0098] FIG. 11 is a flowchart illustrating exemplary operations of a head-worn electronic device for generating a depth map.

[0099] Referring to FIG. 11, in operation 1100, at least one processor (210) may, after acquiring a first frame image, acquire a second frame image (e.g., the second frame image (1210) of FIG. 12) following (or immediately following, or the frame image following the first frame image) of the first FOV through one or more cameras (230). For example, the second frame image may include other visual objects corresponding to objects within the first FOV. Based on the second frame image, at least one processor (210) may acquire fourth depth values ​​of objects within the second FOV through one or more sensors (240).

[0100] According to one embodiment, at least one processor (210) can perform re-projection of the first depth values. At least one processor (210) can re-project the first depth values ​​onto another visual object corresponding to an object in the second frame image. For example, at least one processor (210) can re-project the first depth values ​​onto another visual object corresponding to an object in the second frame image by performing image warping. For example, image warping can be re-projected within the second frame image by changing the position of the first depth values ​​in the first frame image using a position transformation function. For example, at least one processor (210) can accumulate the first depth values ​​onto another visual object in the second frame image by performing image warping. For example, re-projection can be described as converting a three-dimensional image into a two-dimensional image using information from a three-dimensional image. The information of the three-dimensional image may include the coordinates and colors of objects within the three-dimensional image. For example, re-projection may be described as reconstructing the three-dimensional image into a two-dimensional image using the first depth values ​​of the first frame image. For example, re-projection may include converting the two-dimensional image into a three-dimensional image using the information of the two-dimensional image.

[0101] According to one embodiment, at least one processor (210) can determine depth values ​​of an object located outside the second FOV of the second frame image as first depth values. At least one processor (210) can determine depth values ​​of an object located inside the second FOV of the second frame image as fourth depth values ​​and first depth values.

[0102] According to one embodiment, in operation 1110, at least one processor (210) can generate a depth map for a second frame image using first depth values, fourth depth values, and a second frame image. At least one processor (210) can generate a depth map for a second frame image based on the first depth values ​​for an object located outside the second FOV. At least one processor (210) can generate a depth map for a second frame image based on the first depth values ​​and the fourth depth values ​​for an object located inside the second FOV of the second frame image.

[0103] According to one embodiment, at least one processor (210) can generate a 3D mesh (or 3D spot mesh) using first depth values ​​and fourth depth values. The 3D mesh (or 3D spot mesh) may include a truncated signed distance field (TSDF) volume, a surfel-based representation, and / or a spherical mesh. At least one processor (210) can generate a depth map for a second frame image using the 3D mesh (or 3D spot mesh) generated using first depth values ​​and fourth depth values. For example, at least one processor (210) may further utilize a reconstructed mesh of the actual surrounding environment to generate a depth map for the second frame image. A reconstructed mesh of the actual surrounding environment may be stored (or accumulated) in advance within the head-wearing electronic device (200) or may be generated while the head-wearing electronic device (200) is in use. For example, at least one processor (210) may identify a stationary (or immobile) object through the reconstructed mesh while generating a depth map for a second frame image, and may generate a depth map based further on information about said identified object. Generating a depth map for a second frame image is exemplified within the description of FIG. 12.

[0104] FIG. 12 illustrates an example of generating a depth map for a second frame image using depth values ​​obtained based on a first frame image.

[0105] Referring to FIG. 12, at least one processor (210) can acquire a first frame image (400) through one or more cameras (230). At least one processor (210) can identify first depth values ​​of objects within a second FOV through one or more sensors (240) based on the first frame image (400). According to one embodiment, depth values ​​within the second FOV can be acquired while the first frame image is being acquired. Objects within the second FOV may correspond to visual objects (415) within the area (405) of the second FOV of the first frame image (400). The first frame image (400) may include measurement points (1200, 1205) within the area (405) of the second FOV. At least one processor (210) can obtain first depth values ​​of objects corresponding to visual objects (415) based on measurement points (1200, 1205).

[0106] According to one embodiment, at least one processor (210) may decide not to store first depth values ​​of an object corresponding to a visual object (415-1, 415-2) (e.g., first depth values ​​of measurement points (1205)) according to criteria illustrated in the description of FIGS. 5 through 10. At least one processor (210) may decide to store (or accumulate) first depth values ​​of an object corresponding to a visual object (415-3, 415-4) (e.g., first depth values ​​of measurement points (1200)) according to criteria illustrated in the description of FIGS. 5 through 10.

[0107] According to one embodiment, at least one processor (210) may apply first depth values ​​of an object corresponding to a stored (or accumulated) visual object (415-3, 415-4) (e.g., first depth values ​​of measurement points (1200)) to a second frame image (1210). At least one processor (210) may apply the first depth values ​​of an object corresponding to a visual object (415-3, 415-4) (e.g., first depth values ​​of measurement points (1200)) to another visual object (1225-3, 1225-4) corresponding to a visual object (415-3, 415-4) within the second frame image (1210). At least one processor (210) can use the first depth values ​​to determine the depth values ​​of other visual objects (1225-3, 1225-4) (or at least a part of other visual objects (1225-3, 1225-4)) outside the area (1215) of the second FOV of the second frame image (1210). At least one processor (210) can use the first depth values ​​to determine the depth values ​​of objects located outside the second FOV that cannot be acquired while acquiring the second frame image (1210).

[0108] According to one embodiment, at least one processor (210) can obtain fourth depth values ​​of an object within the second FOV through one or more sensors (240) based on the second frame image (1210). The objects within the second FOV may correspond to other visual objects (1225) within the area (1215) of the second FOV of the second frame image (1210). At least one processor (210) can obtain fourth depth values ​​of an object corresponding to the other visual objects (1225-1, 1225-2, 1225-3, 1225-4) based on measurement points (1220) on the other visual objects (1225-1, 1225-2, 1225-3, 1225-4). At least one processor (210) can determine depth values ​​of an object (e.g., an object corresponding to another visual object (1225-3, 1225-4)) in the second FOV based on fourth depth values ​​(e.g., fourth depth values ​​of measurement points (1220) on another visual object (1225-3, 1225-4)) and first depth values ​​(e.g., first depth values ​​of measurement points (1200) within the area (1215) of the second FOV). At least one processor (210) can obtain depth values ​​of an object in the second FOV using a relatively large number of depth values ​​(e.g., first depth values ​​and fourth depth values) by further utilizing the first depth values. At least one processor (210) can determine objects that do not correspond to a reference object (e.g., objects corresponding to other visual objects (1225-3, 1225-4)) as fixed objects. The fourth depth values ​​of objects that do not correspond to a reference object (e.g., objects corresponding to other visual objects (1225-3, 1225-4)) can be used as depth values ​​with high reliability.

[0109] According to one embodiment, at least one processor (210) can generate a depth map for a second frame image (1215) using first depth values ​​(e.g., first depth values ​​of measurement points (1200)), fourth depth values ​​(e.g., fourth depth values ​​of measurement points (1220)), and a second frame image (1215). For example, to generate a depth map for the second frame image (1215), the first depth values ​​of the first frame image (400) may be used (or accumulated, or merged). For example, the first depth values ​​may be further used (or accumulated, or merged) to generate a depth map for a third frame image following the second frame image (1215). A depth map for a second frame image (1215) generated using first depth values ​​(e.g., first depth values ​​of measurement points (1200)) may have depth values ​​for a relatively wider area than other depth maps for a second frame image (1215) generated using the second frame image (1215), and may have relatively accurate depth values. For example, at least one processor (210) may generate a depth map for a second frame image (1215) by performing depth estimation (or depth completion) using first depth values ​​(e.g., first depth values ​​of measurement points (1200)) and fourth depth values ​​(e.g., fourth depth values ​​of measurement points (1220)). For example, at least one processor (210) can use a depth map for the second frame image (1215) to provide a virtual space or perform a three-dimensional reconstruction.For example, at least one processor (210) can provide plane detection, light estimation, and / or spatial audio functions using a depth map for the second frame image (1215).

[0110] FIG. 13 is a block diagram of an electronic device in a network environment according to various embodiments.

[0111] Referring to FIG. 13, in a network environment (1300), an electronic device (1301) may communicate with an electronic device (1302) through a first network (1398) (e.g., a short-range wireless communication network) or with at least one of an electronic device (1304) or a server (1308) through a second network (1399) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (1301) may communicate with the electronic device (1304) through a server (1308). According to one embodiment, the electronic device (1301) may include a processor (1320), memory (1330), input module (1350), sound output module (1355), display module (1360), audio module (1370), sensor module (1376), interface (1377), connection terminal (1378), haptic module (1379), camera module (1380), power management module (1388), battery (1389), communication module (1390), subscriber identification module (1396), or antenna module (1397). In some embodiments, at least one of these components (e.g., connection terminal (1378)) may be omitted from the electronic device (1301), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (1376), camera module (1380), or antenna module (1397)) may be integrated into a single component (e.g., display module (1360)).

[0112] The processor (1320) can, for example, execute software (e.g., program (1340)) to control at least one other component (e.g., hardware or software component) of the electronic device (1301) connected to the processor (1320) and perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (1320) can store commands or data received from other components (e.g., sensor module (1376) or communication module (1390)) in volatile memory (1332), process the commands or data stored in volatile memory (1332), and store the resulting data in non-volatile memory (1334). According to one embodiment, the processor (1320) may include a main processor (1321) (e.g., a central processing unit or an application processor) or an auxiliary processor (1323) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (1301) includes a main processor (1321) and an auxiliary processor (1323), the auxiliary processor (1323) may be configured to use less power than the main processor (1321) or to be specialized for a specified function. The auxiliary processor (1323) may be implemented separately from the main processor (1321) or as part thereof.

[0113] The auxiliary processor (1323) may control at least some of the functions or states associated with at least one component of the electronic device (1301) (e.g., display module (1360), sensor module (1376), or communication module (1390)) on behalf of the main processor (1321) while the main processor (1321) is in an inactive (e.g., sleep) state, or together with the main processor (1321) while the main processor (1321) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (1323) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (1380) or communication module (1390)). According to one embodiment, the auxiliary processor (1323) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (1301) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (1308)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0114] The memory (1330) can store various data used by at least one component of the electronic device (1301) (e.g., processor (1320) or sensor module (1376)). The data may include, for example, input data or output data for software (e.g., program (1340)) and related commands. The memory (1330) may include volatile memory (1332) or non-volatile memory (1334).

[0115] The program (1340) may be stored as software in memory (1330) and may include, for example, an operating system (1342), middleware (1344), or an application (1346).

[0116] The input module (1350) can receive commands or data to be used for a component of the electronic device (1301) (e.g., processor (1320)) from outside the electronic device (1301) (e.g., user). The input module (1350) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0117] The sound output module (1355) can output a sound signal to the outside of the electronic device (1301). The sound output module (1355) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0118] The display module (1360) can visually provide information to an external (e.g., user) of the electronic device (1301). The display module (1360) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (1360) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0119] The audio module (1370) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (1370) can acquire sound through the input module (1350) or output sound through the sound output module (1355) or an external electronic device (e.g., electronic device (1302)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (1301).

[0120] The sensor module (1376) can detect the operating state of the electronic device (1301) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (1376) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0121] The interface (1377) may support one or more specified protocols that can be used for the electronic device (1301) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (1302)). According to one embodiment, the interface (1377) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0122] The connection terminal (1378) may include a connector through which the electronic device (1301) can be physically connected to an external electronic device (e.g., electronic device (1302)). According to one embodiment, the connection terminal (1378) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0123] The haptic module (1379) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (1379) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0124] The camera module (1380) can capture still images and video. According to one embodiment, the camera module (1380) may include one or more lenses, image sensors, image signal processors, or flashes.

[0125] The power management module (1388) can manage the power supplied to the electronic device (1301). According to one embodiment, the power management module (1388) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0126] The battery (1389) can supply power to at least one component of the electronic device (1301). According to one embodiment, the battery (1389) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0127] The communication module (1390) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (1301) and an external electronic device (e.g., electronic device (1302), electronic device (1304), or server (1308)), and the performance of communication through the established communication channel. The communication module (1390) may include one or more communication processors that operate independently of the processor (1320) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (1390) may include a wireless communication module (1392) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (1394) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (1304) through a first network (1398) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (1399) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (1392) can identify or authenticate the electronic device (1301) within a communication network such as the first network (1398) or the second network (1399) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (1396).

[0128] The wireless communication module (1392) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (1392) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (1392) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (1392) can support various requirements specified in the electronic device (1301), external electronic device (e.g., electronic device (1304)), or network system (e.g., second network (1399)). According to one embodiment, the wireless communication module (1392) can support a peak data rate for eMBB realization (e.g., 20 Gbps or more), loss coverage for mMTC realization (e.g., 134 dB or less), or U-plane latency for URLLC realization (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less).

[0129] An antenna module (1397) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (1397) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (1397) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (1398) or a second network (1399), may be selected from the plurality of antennas, for example, by a communication module (1390). A signal or power may be transmitted or received between the communication module (1390) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (1397).

[0130] According to various embodiments, the antenna module (1397) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0131] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0132] According to one embodiment, commands or data may be transmitted or received between the electronic device (1301) and an external electronic device (1304) through a server (1308) connected to a second network (1399). Each of the external electronic devices (1302, or 1304) may be the same or a different type of device as the electronic device (1301). According to one embodiment, all or part of the operations performed on the electronic device (1301) may be performed on one or more of the external electronic devices (1302, 1304, or 1308). For example, if the electronic device (1301) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (1301) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (1301). The electronic device (1301) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (1301) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (1304) may include an Internet of Things (IoT) device. The server (1308) may be an intelligent server using machine learning and / or neural networks.According to one embodiment, an external electronic device (1304) or server (1308) may be included within the second network (1399). The electronic device (1301) may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0133] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0134] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0135] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0136] Various embodiments of the present document may be implemented as software (e.g., program (1340)) comprising one or more instructions stored in a storage medium (e.g., internal memory (1336) or external memory (1338)) readable by a machine (e.g., electronic device (1301)). For example, a processor (e.g., processor (1320)) of the machine (e.g., electronic device (1301)) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0137] According to one embodiment, the method according to the various embodiments disclosed herein may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0138] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0139] FIG. 14a shows an example of a perspective view of a wearable device.

[0140] FIG. 14b shows an example of one or more hardware components placed within a wearable device.

[0141] FIG. 14a illustrates an example of a perspective view of a wearable device. FIG. 14b illustrates an example of one or more hardware components disposed within the wearable device. According to one embodiment, a head-wearing electronic device (200) may have the form of glasses that are wearable on a part of a user's body (e.g., head). The head-wearing electronic device (200) of FIG. 14a and FIG. 14b may be an example of the head-wearing electronic device (200) of FIG. 2. The head-wearing electronic device (200) may include a head-mounted display (HMD). For example, the housing of the head-wearing electronic device (200) may include a flexible material such as rubber and / or silicone that has a shape that adheres to a part of the user's head (e.g., a part of the face covering both eyes). For example, the housing of the head-worn electronic device (200) may include one or more straps that can be twined around the user's head, and / or one or more temples that can be attached to the ears of the head.

[0142] Referring to FIG. 14a, a head-wearing electronic device (200) according to one embodiment may include at least one display (1450) and a frame (1400) supporting at least one display (1450).

[0143] According to one embodiment, a head-worn electronic device (200) may be worn on a part of a user's body. The head-worn electronic device (200) may provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to a user wearing the head-worn electronic device (200). For example, the head-worn electronic device (200) may display a virtual reality image provided by at least one optical device (1482, 1484) of FIG. 14b on at least one display (1450) in response to a specified gesture of the user obtained through the motion recognition camera (1460-2, 1460-3) of FIG. 14b.

[0144] According to one embodiment, at least one display (1450) can provide visual information to a user. For example, at least one display (1450) may include a transparent or translucent lens. At least one display (1450) may include a first display (1450-1) and / or a second display (1450-2) spaced apart from the first display (1450-1). For example, the first display (1450-1) and the second display (1450-2) may be positioned at locations corresponding to the user's left eye and right eye, respectively.

[0145] Referring to FIG. 14b, at least one display (1450) may provide visual information transmitted from external light to a user through a lens included in at least one display (1450) and other visual information distinct from said visual information. The lens may be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, at least one display (1450) may include a first surface (1431) and a second surface (1432) opposite to the first surface (1431). A display area may be formed on the second surface (1432) of at least one display (1450). When a user wears the head-worn electronic device (200), external light may be transmitted to the user by being incident on the first surface (1431) and transmitted through the second surface (1432). As another example, at least one display (1450) can display an augmented reality image combined with a virtual reality image provided by at least one optical device (1482, 1484) on a real image transmitted through external light in a display area formed on a second surface (1432).

[0146] In one embodiment, at least one display (1450) may include at least one waveguide (1433, 1434) that diffracts light emitted from at least one optical device (1482, 1484) and transmits it to a user. At least one waveguide (1433, 1434) may be formed based on at least one of glass, plastic, or polymer. A nano pattern may be formed on the exterior or at least a portion of the interior of at least one waveguide (1433, 1434). The nano pattern may be formed based on a polygonal and / or curved grating structure. Light incident on one end of at least one waveguide (1433, 1434) may be propagated to the other end of at least one waveguide (1433, 1434) by the nano pattern. At least one waveguide (1433, 1434) may include at least one diffractive element (e.g., DOE (diffractive optical element), HOE (holographic optical element)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (1433, 1434) may be placed within a head-worn electronic device (200) to guide a screen displayed by at least one display (1450) to the user's eye. For example, the screen may be transmitted to the user's eye based on total internal reflection (TIR) ​​occurring within at least one waveguide (1433, 1434).

[0147] The head-wearing electronic device (200) can analyze objects included in real-world images collected through a camera (1460-4), combine virtual objects corresponding to objects among the analyzed objects that are the target of augmented reality provision, and display them on at least one display (1450). The virtual object may include at least one of text and images regarding various information related to the objects included in the real-world images. The head-wearing electronic device (200) can analyze objects based on a multi-camera such as a stereo camera. For the object analysis, the head-wearing electronic device (200) can perform spatial recognition (e.g., SLAM (simultaneous localization and mapping)) using a multi-camera and / or time-of-flight (ToF). A user wearing the head-wearing electronic device (200) can view images displayed on at least one display (1450).

[0148] According to one embodiment, the frame (1400) may be formed as a physical structure that allows the head-worn electronic device (200) to be worn on the user's body. According to one embodiment, the frame (1400) may be configured such that when the user wears the head-worn electronic device (200), the first display (1450-1) and the second display (1450-2) can be positioned corresponding to the user's left and right eyes. The frame (1400) may support at least one display (1450). For example, the frame (1400) may support the first display (1450-1) and the second display (1450-2) so that they are positioned corresponding to the user's left and right eyes.

[0149] Referring to FIG. 14a, the frame (1400) may include an area (1420) in which at least a portion of the frame contacts a portion of the user's body when the user wears the head-worn electronic device (200). For example, the area (1420) of the frame (1400) in contact with a portion of the user's body may include an area in contact with a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face that the head-worn electronic device (200) contacts. According to one embodiment, the frame (1400) may include a nose pad (1410) that contacts a portion of the user's body. When the head-worn electronic device (200) is worn by the user, the nose pad (1410) may contact a portion of the user's nose. The frame (1400) may include a first temple (1404) and a second temple (1405) that come into contact with another part of the user's body that is distinct from the part of the user's body.

[0150] For example, the frame (1400) may include a first rim (1401) covering at least a portion of a first display (1450-1), a second rim (1402) covering at least a portion of a second display (1450-2), a bridge (1403) positioned between the first rim (1401) and the second rim (1402), a first pad (1411) positioned along a portion of the edge of the first rim (1401) from one end of the bridge (1403), a second pad (1412) positioned along a portion of the edge of the second rim (1402) from the other end of the bridge (1403), a first temple (1404) extending from the first rim (1401) and fixed to a portion of the wearer's ear, and a second temple (1405) extending from the second rim (1402) and fixed to a portion of the ear opposite to the ear. The first pad (1411) and the second pad (1412) may come into contact with a part of the user's nose, and the first temple (1404) and the second temple (1405) may come into contact with a part of the user's face and a part of the ear. The temples (1404, 1405) may be rotatably connected to the rim through the hinge units (1406, 1407) of FIG. 14b. The first temple (1404) may be rotatably connected to the first rim (1401) through a first hinge unit (1406) positioned between the first rim (1401) and the first temple (1404). The second temple (1405) may be rotatably connected to the second rim (1402) through a second hinge unit (1407) disposed between the second rim (1402) and the second temple (1405). According to one embodiment, the head-wearing electronic device (200) may identify an external object touching the frame (1400) (e.g., a user's fingertip) and / or a gesture performed by said external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of the surface of the frame (1400).

[0151] According to one embodiment, the head-wearing electronic device (200) may include hardware that performs various functions (e.g., hardware described below based on the block diagram of FIG. 16). For example, the hardware may include a battery module (1470), an antenna module (1475), at least one optical device (1482, 1484), speakers (e.g., speakers (1455-1, 1455-2)), microphones (e.g., microphones (1465-1, 1465-2, 1465-3)), a light-emitting module (not shown), and / or a printed circuit board (1490) (e.g., a printed circuit board). The various hardware may be placed within a frame (1400).

[0152] According to one embodiment, a microphone (e.g., microphones (1465-1, 1465-2, 1465-3)) of a head-wearing electronic device (200) is positioned on at least a portion of a frame (1400) to acquire a sound signal. A first microphone (1465-1) positioned on a bridge (1403), a second microphone (1465-2) positioned on a second rim (1402), and a third microphone (1465-3) positioned on a first rim (1401) are shown in FIG. 14b, but the number and position of the microphones (1465) are not limited to the embodiment of FIG. 14b. If the number of microphones (1465) included in the head-wearing electronic device (200) is two or more, the head-wearing electronic device (200) can identify the direction of a sound signal by using multiple microphones placed on different parts of the frame (1400).

[0153] According to one embodiment, at least one optical device (1482, 1484) may project a virtual object onto at least one display (1450) to provide various image information to a user. For example, at least one optical device (1482, 1484) may be a projector. At least one optical device (1482, 1484) may be disposed adjacent to at least one display (1450) or included within at least one display (1450) as part of at least one display (1450). According to one embodiment, a head-worn electronic device (200) may include a first optical device (1482) corresponding to a first display (1450-1) and a second optical device (1484) corresponding to a second display (1450-2). For example, at least one optical device (1482, 1484) may include a first optical device (1482) positioned at the edge of a first display (1450-1) and a second optical device (1484) positioned at the edge of a second display (1450-2). The first optical device (1482) may transmit light to a first waveguide (1433) positioned on the first display (1450-1), and the second optical device (1484) may transmit light to a second waveguide (1434) positioned on the second display (1450-2).

[0154] In one embodiment, the camera (1460) may include a shooting camera (1460-4), an eye tracking camera (ET CAM) (1460-1), and / or a motion recognition camera (1460-2, 1460-3). The shooting camera (1460-4), the eye tracking camera (1460-1), and the motion recognition camera (1460-2, 1460-3) may be positioned at different locations on the frame (1400) and may perform different functions. The eye tracking camera (1460-1) may output data indicating the position of the eyes or the gaze of a user wearing the head-worn electronic device (200). For example, the head-worn electronic device (200) may detect the gaze from an image containing the user's pupils obtained through the eye tracking camera (1460-1). The head-wearing electronic device (200) can identify an object focused by the user (e.g., a real object, and / or a virtual object) by using the user's gaze acquired through the eye-tracking camera (1460-1). The head-wearing electronic device (200), having identified the focused object, can perform a function (e.g., gaze interaction) for interaction between the user and the focused object. The head-wearing electronic device (200) can represent a portion corresponding to the eyes of an avatar representing the user in a virtual space by using the user's gaze acquired through the eye-tracking camera (1460-1). The head-wearing electronic device (200) can render an image (or screen) displayed on at least one display (1450) based on the position of the user's eyes. For example, the visual quality of a first region associated with the gaze within the image and the visual quality of a second region distinct from the first region (e.g., resolution, brightness, saturation, grayscale, PPI) may differ from each other.The head-worn electronic device (200) can acquire an image having a visual quality of a first region and a visual quality of a second region that matches the user's gaze by using foveated rendering. For example, if the head-worn electronic device (200) supports an iris recognition function, user authentication can be performed based on iris information acquired using an eye-tracking camera (1460-1). An example in which the eye-tracking camera (1460-1) is positioned toward the user's right eye is shown in FIG. 14b, but the embodiment is not limited thereto, and the eye-tracking camera (1460-1) may be positioned alone toward the user's left eye or toward both eyes.

[0155] In one embodiment, the camera (1460-4) can capture a real image or background to be matched with a virtual image in order to implement augmented reality or mixed reality content. The camera (1460-4) can be used to acquire high-resolution images based on HR (high resolution) or PV (photo video). The camera (1460-4) can capture an image of a specific object located at the position viewed by the user and provide the image to at least one display (1450). The at least one display (1450) can display a single image in which information regarding a real image or background including the image of the specific object acquired using the camera (1460-4) and a virtual image provided through at least one optical device (1482, 1484) are superimposed. The head-wearing electronic device (200) can compensate for depth information (e.g., the distance between the head-wearing electronic device (200) and an external object obtained through a depth sensor) using an image obtained through a shooting camera (1460-4). The head-wearing electronic device (200) can perform object recognition using an image obtained through a shooting camera (1460-4). The head-wearing electronic device (200) can perform a function of focusing on an object (or subject) within an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using a shooting camera (1460-4). The head-wearing electronic device (200) can perform a pass-through function to display an image obtained through a shooting camera (1460-4) superimposed on at least a portion of a screen representing a virtual space while displaying a screen representing a virtual space on at least one display (1450).In one embodiment, the camera (1460-4) may be placed on a bridge (1403) positioned between the first rim (1401) and the second rim (1402).

[0156] The eye tracking camera (1460-1) can achieve more realistic augmented reality by tracking the gaze of a user wearing a head-worn electronic device (200), thereby matching the user's gaze with visual information provided to at least one display (1450). For example, the head-worn electronic device (200) can naturally display environmental information related to the user's front at the location where the user is situated on at least one display (1450) when the user looks straight ahead. The eye tracking camera (1460-1) may be configured to capture an image of the user's pupil to determine the user's gaze. For example, the eye tracking camera (1460-1) may receive a gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the eye tracking camera (1460-1) may be positioned at locations corresponding to the user's left and right eyes. For example, the eye-tracking camera (1460-1) may be positioned within the first rim (1401) and / or the second rim (1402) to face the direction in which the user wearing the head-worn electronic device (200) is located.

[0157] A motion recognition camera (1460-2, 1460-3) can provide a specific event to a screen provided on at least one display (1450) by recognizing the movement of the user's entire body or part thereof, such as the user's torso, hands, or face. A motion recognition camera (1460-2, 1460-3) can recognize the user's gesture, acquire a signal corresponding to the gesture, and provide a display corresponding to the signal to at least one display (1450). A processor can identify the signal corresponding to the gesture and, based on the identification, perform a designated function. A motion recognition camera (1460-2, 1460-3) can be used to perform spatial recognition functions using SLAM and / or depth maps for a 6-degrees-of-freedom pose (6 dof pose). A processor can use the motion recognition camera (1460-2, 1460-3) to perform gesture recognition functions and / or object tracking functions. In one embodiment, a motion recognition camera (1460-2, 1460-3) may be placed on the first rim (1401) and / or the second rim (1402).

[0158] The camera (1460) included in the head-worn electronic device (200) is not limited to the eye-tracking camera (1460-1) and motion recognition camera (1460-2, 1460-3) described above. For example, the head-worn electronic device (200) can identify external objects included within the FoV by using a camera positioned toward the user's FoV. The identification of external objects by the head-worn electronic device (200) can be performed based on a sensor for identifying the distance between the head-worn electronic device (200) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (1460) positioned toward the FoV may support an autofocus function and / or an optical image stabilization (OIS) function. For example, the head-worn electronic device (200) may include a camera (1460) (e.g., a face tracking camera) positioned toward the face to acquire an image including the face of a user wearing the head-worn electronic device (200).

[0159] Although not illustrated, according to one embodiment, a head-worn electronic device (200) may further include a light source (e.g., LED) that emits light toward a subject (e.g., user's eyes, face, and / or external objects within the FoV) being photographed using a camera (1460). The light source may include an LED of infrared wavelength. The light source may be placed in at least one of a frame (1400) and hinge units (1406, 1407).

[0160] According to one embodiment, the battery module (1470) can supply power to the electronic components of the head-wearing electronic device (200). In one embodiment, the battery module (1470) may be placed within the first temple (1404) and / or the second temple (1405). For example, the battery module (1470) may be a plurality of battery modules (1470). The plurality of battery modules (1470) may each be placed in the first temple (1404) and the second temple (1405), respectively. In one embodiment, the battery module (1470) may be placed at the end of the first temple (1404) and / or the second temple (1405).

[0161] The antenna module (1475) can transmit a signal or power to the outside of the head-wearing electronic device (200) or receive a signal or power from the outside. In one embodiment, the antenna module (1475) may be placed within the first temple (1404) and / or the second temple (1405). For example, the antenna module (1475) may be placed near one side of the first temple (1404) and / or the second temple (1405).

[0162] A speaker (1455) can output an acoustic signal to the outside of the head-wearing electronic device (200). The acoustic output module may be referred to as a speaker. In one embodiment, the speaker (1455) may be placed within a first temple (1404) and / or a second temple (1405) to be positioned adjacent to the ears of a user wearing the head-wearing electronic device (200). For example, the speaker (1455) may include a second speaker (1455-2) positioned adjacent to the user's left ear by being placed within the first temple (1404), and a first speaker (1455-1) positioned adjacent to the user's right ear by being placed within the second temple (1405).

[0163] A light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the head-wearing electronic device (200) to the user. For example, if the head-wearing electronic device (200) requires charging, it may emit red light at a constant frequency. In one embodiment, the light-emitting module may be placed on the first rim (1401) and / or the second rim (1402).

[0164] Referring to FIG. 14b, a head-wearing electronic device (200) according to one embodiment may include a printed circuit board (PCB) (1490). The PCB (1490) may be included in at least one of a first temple (1404) or a second temple (1405). The PCB (1490) may include an interposer disposed between at least two sub-PCBs. On the PCB (1490), one or more hardware components included in the head-wearing electronic device (200) (e.g., hardware components illustrated by different blocks in FIG. 4) may be disposed. The head-wearing electronic device (200) may include a flexible PCB (FPCB) for interconnecting the hardware components.

[0165] According to one embodiment, a head-worn electronic device (200) may include at least one of a gyroscope sensor, a gravity sensor, and / or an acceleration sensor for detecting the posture of the head-worn electronic device (200) and / or the posture of a body part (e.g., head) of a user wearing the head-worn electronic device (200). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on designated three-dimensional axes (e.g., x-axis, y-axis, and z-axis) that are perpendicular to each other. The gyroscope sensor may measure the angular velocity of each of the designated three-dimensional axes (e.g., x-axis, y-axis, and z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyroscope sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the head-wearing electronic device (200) can identify a user's motion and / or gesture performed to execute or interrupt a specific function of the head-wearing electronic device (200) based on an IMU.

[0166] FIGS. 15a to 15b show an example of the appearance of a wearable device.

[0167] FIGS. 15a and 15b illustrate an example of the appearance of a wearable device (e.g., a head-wearing electronic device (200)). The head-wearing electronic device (200) of FIGS. 15a and 15b may be an example of the head-wearing electronic device (200) of FIG. 2. According to one embodiment, an example of the appearance of a first surface (1510) of the housing of the head-wearing electronic device (200) may be illustrated in FIG. 15a, and an example of the appearance of a second surface (1520) opposite to the first surface (1510) may be illustrated in FIG. 15b.

[0168] Referring to FIG. 15a, according to one embodiment, a first surface (1510) of a head-wearing electronic device (200) may have a shape that is attachable to a part of a user's body (e.g., the face of the user). Although not illustrated, the head-wearing electronic device (200) may further include a strap for securing to a part of a user's body and / or one or more temples (e.g., a first temple (1404) and / or a second temple (1405) of FIG. 14a to FIG. 14b). A first display (1450-1) for outputting an image to the left eye among the user's two eyes, and a second display (1450-2) for outputting an image to the right eye among the two eyes may be disposed on the first surface (1510). The head-wearing electronic device (200) may further include rubber or silicone packing formed on the first surface (1510) to prevent interference by light different from light emitted from the first display (1450-1) and the second display (1450-2) (e.g., ambient light).

[0169] According to one embodiment, a head-worn electronic device (200) may include cameras (1460-1) for photographing and / or tracking both eyes of a user adjacent to each of the first display (1450-1) and the second display (1450-2). The cameras (1460-1) may be referenced to the eye-tracking camera (1460-1) of FIG. 14b. According to one embodiment, a head-worn electronic device (200) may include cameras (1460-5, 1460-6) for photographing and / or recognizing a user's face. The cameras (1460-5, 1460-6) may be referenced to FT cameras. The head-worn electronic device (200) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the cameras (1460-5, 1460-6). For example, the head-worn electronic device (200) can change the texture and / or shape of a part of an avatar (e.g., a part of an avatar representing a human face) by using information obtained by cameras (1460-5, 1460-6) (e.g., FT cameras) and representing the facial expression of a user wearing the head-worn electronic device (200).

[0170] Referring to FIG. 15b, on a second surface (1520) opposite to the first surface (1510) of FIG. 15a, a camera (e.g., cameras (1460-7, 1460-8, 1460-9, 1460-10, 1460-11, 1460-12)), and / or a sensor (e.g., a depth sensor (1530)) may be placed to acquire information related to the external environment of the head-wearing electronic device (200). For example, cameras (1460-7, 1460-8, 1460-9, 1460-10) may be placed on the second surface (1520) to recognize external objects. The cameras (1460-7, 1460-8, 1460-9, 1460-10) can be referenced to the motion recognition cameras (1460-2, 1460-3) of FIG. 14b.

[0171] For example, using cameras (1460-11, 1460-12), the head-wearing electronic device (200) can acquire images and / or videos to be transmitted to each of the user's two eyes. Camera (1460-11) may be placed on the second surface (1520) of the head-wearing electronic device (200) to acquire an image to be displayed through a second display (1450-2) corresponding to the right eye among the two eyes. Camera (1460-12) may be placed on the second surface (1520) of the head-wearing electronic device (200) to acquire an image to be displayed through a first display (1450-1) corresponding to the left eye among the two eyes. Cameras (1460-14, 1460-15) may be referenced to the shooting camera (1460-4) of FIG. 14b.

[0172] According to one embodiment, a head-worn electronic device (200) may include a depth sensor (1530) disposed on a second surface (1520) to identify the distance between the head-worn electronic device (200) and an external object. Using the depth sensor (1530), the head-worn electronic device (200) may acquire spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the head-worn electronic device (200). Although not illustrated, a microphone may be disposed on the second surface (1520) of the head-worn electronic device (200) to acquire sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0173] Hereinafter, with reference to FIG. 16, the hardware or software configuration of the head-worn electronic device (200) is described.

[0174] Figure 16 shows an example of a block diagram of a wearable device.

[0175] FIG. 16 illustrates an example of a block diagram of a wearable device (e.g., a head-worn electronic device (200)). The head-worn electronic device (200) of FIG. 16 may be an example of the head-worn electronic device (200) of FIG. 2 and the head-worn electronic device (200) of FIG. 14a through FIG. 15b.

[0176] Referring to FIG. 16, a head-wearing electronic device (200) according to one embodiment may include a processor (1610), memory (1615), a display (1450) (e.g., a first display (1450-1) and / or a second display (1450-2) of FIG. 14a, FIG. 14b, FIG. 15a, and FIG. 15b), and / or a sensor (1617). The processor (1610), memory (1615), display (1450) and / or sensor (1617) may be electrically and / or operationally connected to each other by an electronic component such as a communication bus (1602). In the present disclosure, the operational connection of the electronic components may include a direct connection established between the electronic components and / or an indirect connection established between the electronic components such that a first electronic component among the electronic components is controlled by a second electronic component among the electronic components. The type and / or number of electronic components included in the head-wearing electronic device (200) are not limited to those shown in FIG. 16. For example, the head-wearing electronic device (200) may include only some of the electronic components shown in FIG. 16.

[0177] A processor (1610) of a head-wearing electronic device (200) according to one embodiment may include a circuit (e.g., a processing circuit) for processing data based on one or more instructions. The circuit for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). In one embodiment, the head-wearing electronic device (200) may include one or more processors. The processor (1610) may have a structure of a multi-core processor such as a dual core, a quad core, a hexa core, and / or an octa core. The multi-core processor structure of the processor (1610) may include a structure based on multiple core circuits (e.g., a big-little structure), distinguished by power consumption, clock, and / or computational power per unit time. In one embodiment comprising a processor (1610) having a multi-core processor structure, the operations and / or functions of the present disclosure may be performed individually or collectively by one or more cores included in the processor (1610).

[0178] A memory (1615) of a head-worn electronic device (200) according to one embodiment may include electronic components for storing data and / or instructions that are input to or output from a processor (1610). The memory (1615) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). Volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). Non-volatile memory may include, for example, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disk, and embedded multimedia card (eMMC). In one embodiment, the memory (1615) may be referred to as storage.

[0179] In one embodiment, a display (1450) of a head-wearing electronic device (200) can output visualized information to a user of the head-wearing electronic device (200). A display (1450) arranged in front of the eyes of a user wearing the head-wearing electronic device (200) may be placed in at least a part of the housing of the head-wearing electronic device (200) (e.g., a first display (1450-1) and / or a second display (1450-2) of FIG. 14a, FIG. 14b, FIG. 15a, and FIG. 15b). For example, the display (1450) may be controlled by a processor (1610) including circuits such as a CPU, a GPU (graphic processing unit), and / or a DPU (display processing unit) to output visualized information to the user. The display (1450) may include a flexible display, a flat panel display (FPD), and / or electronic paper. The display (1450) may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs may include organic LEDs (OLEDs). Embodiments are not limited thereto, for example, if the head-wearing electronic device (200) includes a lens for transmitting external light (or ambient light), the display (1450) may include a projector (or projection assembly) for projecting light onto said lens. In one embodiment, the display (1450) may be referred to as a display panel and / or a display module. Pixels included in the display (1450) may be positioned toward either of the user's two eyes when worn by the user of the head-wearing electronic device (200).For example, the display (1450) may include display areas (or active areas) corresponding to each of the user's two eyes.

[0180] In one embodiment, a sensor (1617) of a head-wearing electronic device (200) may generate electrical information that can be processed by a processor (1610) and / or memory (1615) from non-electronic information related to the head-wearing electronic device (200). For example, the sensor (1617) may include a global positioning system (GPS) sensor for detecting the geographic location of the head-wearing electronic device (200). In addition to the GPS method, the sensor (1617) may generate information indicating the geographic location of the head-wearing electronic device (200) based on a global navigation satellite system (GNSS), such as Galileo or Beidou (compass), for example. The above information may be stored in memory (1615), processed by a processor (1610), and / or transmitted to another electronic device distinct from the head-wearing electronic device (200) via a communication circuit.

[0181] According to one embodiment, within the memory (1615) of the head-wearing electronic device (200), one or more instructions (or commands) representing data to be processed by the processor (1610) of the head-wearing electronic device (200), calculations to be performed, and / or operations may be stored. A set of one or more instructions may be referred to as a program, firmware, operating system, process, routine, sub-routine, and / or software application (hereinafter, application). For example, the head-wearing electronic device (200), and / or processor (1610) may perform at least one of the operations of FIGS. 3 through 12 when a set of a plurality of instructions distributed in the form of an operating system, firmware, driver, program, and / or software application is executed. In the following, the statement that a software application is installed within a head-worn electronic device (200) may mean that one or more instructions provided in the form of a software application (or package) are stored in memory (1615), and that said one or more applications are stored in an executable format (e.g., a file having an extension specified by the operating system of the head-worn electronic device (200)) by the processor (1610). For example, the application may include a program and / or library related to a service provided to a user.

[0182] Referring to FIG. 16, programs installed on a head-worn electronic device (200) may be included in any one of different layers, including an application layer (1640), a framework layer (1650), and / or a hardware abstraction layer (HAL) (1680), depending on the target. For example, within the hardware abstraction layer (1680), programs (e.g., modules, or drivers) designed to target the hardware of the head-worn electronic device (200) (e.g., a display (1450), and / or a sensor (1617)) may be included. The framework layer (1650) may be referred to as an XR framework layer in that it includes one or more programs for providing XR (extended reality) services. For example, the layers illustrated in FIG. 16 are logically (or for convenience of explanation) separated, and this does not mean that the address space of memory (1615) is separated by said layers.

[0183] For example, within the framework layer (1650), programs designed to target at least one of the hardware abstraction layer (1680) and / or the application layer (1640) (e.g., a location tracker (1671), a spatial recognizer (1672), a gesture tracker (1673), an eye tracker (1674), and / or a face tracker (1675)) may be included. The programs included in the framework layer (1650) may provide an application programming interface (API) that is executable (or invokeable) based on other programs.

[0184] For example, within the application layer (1640), a program designed to target a user of the head-worn electronic device (200) may be included. Examples of programs included in the application layer (1640) include an XR (extended reality) system UI (user interface) (1641) and / or an XR application (1642), but embodiments are not limited thereto. For example, programs included in the application layer (1640) (e.g., software applications) may call an API to cause the execution of a function supported by programs included in the framework layer (1650).

[0185] For example, the head-wearing electronic device (200) may display one or more visual objects on the display (1450) to perform interaction with the user based on the execution of the XR system UI (1641). A visual object may mean an object that can be placed on the screen for the transmission of information and / or interaction, such as text, images, icons, videos, buttons, checkboxes, radio buttons, text boxes, sliders, and / or tables. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The head-wearing electronic device (200) may provide the user with functions available in a virtual space based on the execution of the XR system UI (1641).

[0186] Referring to FIG. 16, a lightweight renderer (1643) and / or an XR plugin (1644) are depicted within the XR system UI (1641), but are not limited thereto. For example, based on the XR system UI (1641), the processor (1610) may execute the lightweight renderer (1643) and / or an XR plugin (1644) within the framework layer (1650).

[0187] For example, a head-worn electronic device (200) may acquire resources (e.g., APIs, system processes and / or libraries) used to define, create, and / or execute a rendering pipeline, which is permitted to be partially modified, based on the execution of a lightweight renderer (1643). The lightweight renderer (1643) may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline, which is permitted to be partially modified. The lightweight renderer (1643) may include a renderer built prior to the execution of a software application (e.g., a pre-built renderer). For example, the head-worn electronic device (200) may acquire resources (e.g., APIs, system processes and / or libraries) used to define, create, and / or execute the entire rendering pipeline based on the execution of an XR plugin (1644). The XR plugin (1644) can be referred to as an open XR native client in terms of defining (or setting) the entire rendering pipeline.

[0188] For example, the head-worn electronic device (200) may display a screen representing at least a portion of a virtual space on a display (1450) based on the execution of an XR application (1642). An XR plugin (1644-1) included in the XR application (1642) may include instructions that support functions similar to those of an XR plugin (1644) of an XR system UI (1641). Descriptions of the XR plugin (1644-1) that overlap with descriptions of the XR plugin (1644) may be omitted. The head-worn electronic device (200) may trigger the execution of a virtual space manager (1651) based on the execution of the XR application (1642).

[0189] For example, the head-wearing electronic device (200) may display an image on a display (1450) in virtual space based on the execution of an application (1645). The application (1645) may be configured to output image information for displaying a two-dimensional image. The head-wearing electronic device (200) may trigger the execution of a virtual space manager (1651) based on the execution of the application (1645). The head-wearing electronic device (200) may generate dual image information to display the two-dimensional image in three-dimensional virtual space based on the execution of the application (1645). Here, the dual image information may include a first image information for the left eye and a second image information for the right eye, taking into account binocular parallax. To display the two-dimensional image in three-dimensional virtual space, the head-wearing electronic device (200) may generate the dual image information based on the image information for displaying the two-dimensional image.

[0190] According to one embodiment, the head-worn electronic device (200) may provide virtual space services based on the execution of a virtual space manager (1651). For example, the virtual space manager (1651) may include a platform for supporting virtual space services. Based on the execution of the virtual space manager (1651), the head-worn electronic device (200) may identify a virtual space formed based on the user's location indicated by data acquired through a sensor (1617) and may display at least a portion of the virtual space on a display (1450). The virtual space manager (1651) may be referred to as a composition presentation manager (CPM).

[0191] For example, the virtual space manager (1651) may include a runtime service (1652). For example, the runtime service (1652) may be referred to as an OpenXR runtime module (or OpenXR runtime program). The head-worn electronic device (200) may execute at least one of a user pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (1652). For example, the head-worn electronic device (200) may perform rendering for a virtual space service for the user based on the execution of the runtime service (1652). For example, a virtual space-related function executable by the application layer (1640) may be supported based on the execution of the runtime service (1652).

[0192] For example, the virtual space manager (1651) may include a pass-through manager (1653). The head-worn electronic device (200), based on the execution of the pass-through manager (1653), may display an image and / or video representing the real space acquired through an external camera superimposed on at least a portion of the screen while displaying a screen representing the virtual space on the display (1450).

[0193] For example, the virtual space manager (1651) may include an input manager (1654). The head-worn electronic device (200) may identify acquired data (e.g., sensor data) by executing one or more programs included within the recognition service layer (1670) based on the execution of the input manager (1654). The head-worn electronic device (200) may identify user inputs associated with the head-worn electronic device (200) using the acquired data. The user inputs may be associated with user motions (e.g., hand gestures), gaze, and / or speech identified by a sensor (1617) (e.g., an image sensor such as an external camera). The user inputs may be identified based on an external electronic device connected (or paired) via a communication circuit.

[0194] For example, the perception abstract layer (1660) can be used for data exchange between the virtual space manager (1651) and the perception service layer (1670). In terms of being used for data exchange between the virtual space manager (1651) and the perception service layer (1670), the perception abstract layer (1660) can be referred to as an interface. As an example, the perception abstract layer (1660) can be referred to as OpenPX. The perception abstract layer (1660) can be used for a perception client and a perception service.

[0195] According to one embodiment, the recognition service layer (1670) may include one or more programs for processing data obtained from the sensor (1617). The one or more programs may include at least one of a location tracker (1671), a spatial recognizer (1672), a gesture tracker (1673), and / or an eye tracker (1674). The type and / or number of the one or more programs included in the recognition service layer (1670) are not limited to those shown in FIG. 16.

[0196] For example, the head-wearing electronic device (200) can identify the posture of the head-wearing electronic device (200) using a sensor based on the execution of the position tracker (1671). The head-wearing electronic device (200) can identify the 6 degrees of freedom pose (6 dof pose) of the head-wearing electronic device (200) using data acquired using an external camera (e.g., an image sensor) and / or an IMU (e.g., a motion sensor including a gyroscope, an accelerometer, and / or a geomagnetic sensor) based on the execution of the position tracker (1671). The position tracker (1671) may be referred to as a head tracking (HeT) module (or head tracker, head tracking program).

[0197] For example, the head-wearing electronic device (200) may acquire information to provide a three-dimensional virtual space corresponding to the surrounding environment (e.g., external space) of the head-wearing electronic device (200) (or the user of the head-wearing electronic device (200)) based on the execution of the spatial recognition device (1672). The head-wearing electronic device (200) may recreate the surrounding environment of the head-wearing electronic device (200) in three dimensions using data acquired using an external camera (e.g., image sensor) based on the execution of the spatial recognition device (1672). The head-wearing electronic device (200) may identify at least one of a plane, a slope, and a staircase based on the surrounding environment of the head-wearing electronic device (200) recreated in three dimensions based on the execution of the spatial recognition device (1672). The spatial recognition device (1672) may be referred to as a scene understanding (SU) module (or scene understanding program).

[0198] For example, the head-wearing electronic device (200) can identify (or recognize) the pose and / or gesture of the user's hand of the head-wearing electronic device (200) based on the execution of the gesture tracker (1673). For example, the head-wearing electronic device (200) can identify the pose and / or gesture of the user's hand using data acquired from an external camera (e.g., an image sensor) based on the execution of the gesture tracker (1673). For example, the head-wearing electronic device (200) can identify the pose and / or gesture of the user's hand based on data (or images) acquired using an external camera based on the execution of the gesture tracker (1673). The gesture tracker (1673) may be referred to as a hand tracking (HaT) module (or hand tracking program) and / or a gesture tracking module.

[0199] For example, the head-wearing electronic device (200) can identify (or track) the movement of the user's eyes of the head-wearing electronic device (200) based on the execution of the eye tracker (1674). For example, the head-wearing electronic device (200) can identify the movement of the user's eyes using data obtained from an eye-tracking camera (e.g., an image sensor) based on the execution of the eye tracker (1674). The eye tracker (1674) may be referred to as an eye tracking (ET) module (or eye tracking program) and / or a gaze tracking module.

[0200] For example, the recognition service layer (1670) of the head-wearing electronic device (200) may further include a face tracker (1675) for tracking the user's face. For example, the head-wearing electronic device (200) may identify (or track) the movement of the user's face and / or the user's facial expression based on the execution of the face tracker (1675). The head-wearing electronic device (200) may estimate the user's facial expression based on the movement of the user's face based on the execution of the face tracker (1675). For example, the head-wearing electronic device (200) may identify the movement of the user's face and / or the user's facial expression based on data (e.g., images and / or videos) acquired using a camera (1625) (e.g., a camera facing at least a part of the user's face) based on the execution of the face tracker (1675).

[0201] Referring to FIG. 16, the renderer (1690) may include instructions for rendering images in a three-dimensional virtual space. A processor (1610) that executes the renderer (1690) may obtain at least one image to be displayed at least partially in a display area of ​​the display (1450) in a software application. For example, the processor (1610) that executes the renderer (1690) may determine the location of the area where an application (e.g., XR application (1642), application (1645)) will be rendered. The processor (1610) that executes the renderer (1690) may generate an image of said application to be displayed on the display (1450). The renderer (1690) may synthesize images to generate a composite image to be displayed on the display (1450).

[0202] For example, a processor (1610) that executes a renderer (1690) can divide the display area of ​​a display (1450) into a foveated portion (or may be referred to as a foveated area) and a peripheral portion (or may be referred to as a residual area) using a gaze position calculated using a position tracker (1671) and / or a gaze tracker (1674). For example, a processor (1610) that detects coordinate values ​​of the gaze position can determine the portion of the display area containing said coordinate values ​​as the foveated area. A DPU that executes a renderer (1690) can acquire at least one image corresponding to each of said foveated area and said residual area, having a size smaller than the size of the entire display area of ​​the display (1450) or having a resolution less than the resolution of the display area.

[0203] A processor (1610) that executes a renderer (1690) can obtain or generate a composite image to be displayed on a display (1450) by synthesizing an image corresponding to a foveated area and an image corresponding to a surrounding area. For example, the processor (1610) can perform upscaling to enlarge the image corresponding to the surrounding area to the size of the entire display area of ​​the display (1450). On the enlarged image, the processor (1610) can combine the image corresponding to the foveated area to generate a composite image to be displayed on the display (1450). Along the boundary line of the image corresponding to the foveated area, the processor (1610) can mix the enlarged image and the image corresponding to the foveated area by applying a visual effect such as blur.

[0204] Figure 17 shows an example of a block diagram of an electronic device for displaying an image in virtual space.

[0205] In FIG. 17, an example is described in which multiple programs / instructions are executed to display an image in a virtual space. The multiple programs / instructions may all be executed on a single processor (e.g., AP) or may be executed by multiple processors (e.g., AP, GPU (graphic processing unit), NPU (neural processing unit)). The meaning of being able to be executed by multiple processors is that some programs / instructions may be executed by a first processor and other programs / instructions may be executed by a second processor different from the first processor.

[0206] Referring to FIG. 17, a head-worn electronic device (200) may execute a virtual space manager (1750) (e.g., the virtual space manager (1651) of FIG. 16, CPM) to render an image in a virtual space. For the virtual space manager (1750), at least some of the descriptions of the virtual space manager (1651) of FIG. 16 may be referenced. The virtual space manager (1750) may include a platform for supporting virtual space services. The virtual space manager (1750) may include a runtime service (1751) (e.g., open XR runtime), a panel renderer (1752) (e.g., 2D panel render), and an XR compositor (1753) (XR compositor). Based on the execution of the runtime service (1751), the head-worn electronic device (200) may execute at least one of a user pose prediction function, a frame timing function, and / or a spatial input function. For the runtime service (1751), at least some of the descriptions of the runtime service (1652) of FIG. 16 may be referenced. The head-wearing electronic device (200) may display at least one image (video) on a panel (e.g., a 2D panel) to enable the implementation of a virtual space through a display based on the execution of panel rendering (1752). For example, the head-wearing electronic device (200) may display a rendering image corresponding to RGB information (1766) for a panel from the spatialization manager (1740) described below through a display (e.g., a display (1750)). The head-wearing electronic device (200) may composite an image of a real area (hereinafter, a pass-through image) captured through a camera in virtual space with an image of a virtual area based on the execution of an XR compositor (1753) (XR compositor). For example, the head-worn electronic device (200) can generate a composite image by merging the pass-through image and the virtual region image based on the execution of the XR synthesis unit (1753).The head-wearing electronic device (200) can transmit the generated composite image to a display buffer so that the composite image is displayed. The head-wearing electronic device (200) can identify a virtual space through a virtual space manager (1750) and can display at least a portion of the virtual space on the display (1750). The virtual space manager (1750) may be referred to as a CPM. The head-wearing electronic device (200) can execute the virtual space manager (1750) to render an image corresponding to at least a portion of the virtual space.

[0207] According to one embodiment, a head-worn electronic device (200) may execute a spatialization manager (1740). The spatialization manager (1740) may perform processing for displaying an image in a three-dimensional virtual space. The head-worn electronic device (200) may perform preprocessing based on the execution of the spatialization manager (1740) so that an image can be rendered in a three-dimensional virtual space through a virtual space manager (1750). For example, the head-worn electronic device (200) may perform at least some of the functions of the renderer (1690) of FIG. 16 based on the execution of the spatialization manager (1740). The head-worn electronic device (200) may process image information provided by an application (e.g., an XR application (1710), an application providing a non-XR general 2D screen (1720), an application providing a system UI (1730)) based on the execution of the spatialization manager (1740). A spatialization manager (1740) (e.g., space flinger) may include a system screen manager (1741) (e.g., system scene), an input manager (1742) (e.g., input routing), and a lightweight rendering engine (1743) (e.g., impress engine). The system screen manager (1741) may be executed to display a system UI (1730). System UI-related information (1764) may be transmitted to the system screen manager (1741) from a program (e.g., API) that provides the system UI (1730). System UI-related information (1764) may be obtained through a spatializer API and / or a same-process private API. The spatialization manager (1740) may determine the layout (e.g., position, display order) of the system UI (1730) screen in three-dimensional space through pre-allocated resources.The system screen manager (1741) may transmit image information (1767) to the virtual space manager (1750) for rendering a screen of the system UI (1730) according to the layout. The input manager (1742) may be configured to process user input (e.g., user input on a system screen or app screen). The Impress engine may be a renderer for image generation (e.g., a lightweight renderer (1743)). For example, the Impress engine may be used to display the system UI (1730). According to one embodiment, the spatialization manager (1740) may include a lightweight rendering engine (1743) for rendering the system UI. According to one embodiment, if the lightweight rendering engine (1743) does not have sufficient resources to render an avatar used in an HMD, at least one external rendering engine may be used. At this time, to resolve compatibility issues with external rendering (e.g., 3rd party engine), an external rendering engine support module may be added inside the spatialization manager (1740).

[0208] According to one embodiment, the electronic device may execute an application. For example, in response to the execution of an XR application (1710) (e.g., XR application (1710), 3D game, XR map, other immersive application), the virtual space manager (1750) may be executed. The head-worn electronic device (200) may provide dual image information (1761) provided from the XR application (1710) to the virtual space manager (1750). To display images in three-dimensional space, the dual image information (1761) may include two image information that account for binocular parallax. For example, the dual image information (1761) may include a first image information for the user's left eye and a second image information for the user's right eye to render in three-dimensional virtual space. Hereinafter, the term dual image information is used in the present disclosure to refer to image information for displaying images for both eyes in three-dimensional space. In addition to the dual image information, the above dual image information may utilize binocular image information, dual image information, dual image data, dual image, binocular image data, stereoscopic image information, 3D image information, spatial image information, spatial image data, 2D-3D conversion data, dimension conversion image data, binocular parallax image data, and / or equivalent technical terms. The head-wearing electronic device (200) can generate a composite image by merging image layers through a virtual space manager (1750). The head-wearing electronic device (200) can transmit the generated composite image to a display buffer. The composite image can be displayed on the display (1750) of the head-wearing electronic device (200).

[0209] According to one embodiment, the electronic device may execute at least one application among an XR application (1710) and other applications (1720) (e.g., a first application (1720-1), a second application (1720-2), ..., a Nth application (1720-N)). According to one embodiment, the application (1720) may be configured to output image information for displaying a two-dimensional image. In other words, the application (1720) may provide a two-dimensional image. As an example, the application (1720) may be a video application, a schedule application, or an internet browser application. Let us assume that, in response to the execution of the application (1720), image information (1762) provided from the application (1720) is provided to a virtual space manager (1750). Since the image information (1762) has only x and y coordinates within a two-dimensional plane, it may be difficult to consider the order of other applications centered on the user (i.e., distance from the user). The head-worn electronic device (200) may execute a spatialization manager (1740) to provide dual image information to a virtual space manager (1750), even when displaying an application (1720) that provides a general 2D screen. For example, based on the execution of the spatialization manager (1740), the head-worn electronic device (200) may receive application-related information (1763) from the first application (1720-1). For example, application-related information (1763) may include image information representing a two-dimensional image of the first application (1720-1) (e.g., information including RGB per pixel) and / or content information in the first application (1720-1) (e.g., characteristics of content running in the first application, type of content). Application-related information (1763) may be obtained through a spatializer API.Based on the execution of the spatialization manager (1740), the head-wearing electronic device (200) can identify information regarding the location of the area to be rendered and the size of the area to be rendered (hereinafter, location information). Based on the execution of the spatialization manager (1740), the head-wearing electronic device (200) can generate dual image information (1765, e.g., RGBx2) that takes into account the user's binocular parallax through the image information and the location information. Based on the execution of the spatialization manager (1740), the head-wearing electronic device (200) can provide the dual image information (1765) to the virtual space manager (1750). By converting a simple two-dimensional image into dual image information (1765), the problem caused by the image information (1762) being directly transmitted to the virtual space manager (1750) can be resolved. Additionally, as at least some of the functions for displaying images in virtual space are performed by the spatialization manager (1740) instead of the virtual space manager (1750), the burden on the virtual space manager (1750) may be reduced.

[0210] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains.

[0211] The head-wearing electronic device described above (e.g., the head-wearing electronic device (200) of FIG. 2) may include at least one processor (e.g., at least one processor (210) of FIG. 2) having a processing circuit, one or more cameras having a first field of view (FOV) (e.g., one or more cameras (230) of FIG. 2), one or more sensors having a second field of view narrower than the first FOV (e.g., one or more sensors (240) of FIG. 2), and a memory (e.g., memory (220) of FIG. 2) including one or more storage media configured to store one or more programs configured to be executed individually or collectively by the at least one processor. The one or more programs may include instructions that cause the head-wearing electronic device to acquire a first frame image of the first FOV of the one or more cameras (e.g., the first frame image (400) of FIG. 4) through the one or more cameras. The above one or more programs may include instructions that cause a head-worn electronic device to identify an object located within the second FOV of the one or more sensors corresponding to a representation in the first frame image. The above one or more programs may include instructions that cause a head-worn electronic device to identify whether the object corresponds to a reference object. The above one or more programs may include instructions that cause a head-worn electronic device to store depth values ​​obtained through the one or more sensors with respect to the object while acquiring the first frame image in order to generate a depth map for a second frame image (e.g., the second frame image (1210) of FIG. 12) to be acquired through the one or more cameras based on identifying that the object is different from the reference object.The above one or more programs may include instructions that cause a head-worn electronic device to refrain from storing the depth values ​​by discarding the depth values ​​based on identifying that the object corresponds to the reference object.

[0212] For example, the depth values ​​may be first depth values. The one or more programs may include instructions that cause a head-worn electronic device to identify a distorted region within the representation within the first frame image based on identifying that the object is different from the reference object. The one or more programs may include instructions that cause a head-worn electronic device to store second depth values ​​of the undistorted region within the representation among the first depth values. The one or more programs may include instructions that cause a head-worn electronic device to refrain from storing the third depth values ​​by discarding the third depth values ​​of the distorted region within the representation among the first depth values.

[0213] For example, the depth values ​​may be first depth values. The one or more programs may include instructions that cause a head-worn electronic device to identify confidence values ​​for each of the first depth values ​​based on identifying that the object is different from the reference object. The one or more programs may include instructions that cause a head-worn electronic device to store second depth values ​​among the first depth values ​​that have a confidence value greater than or equal to the reference confidence value. The one or more programs may include instructions that cause a head-worn electronic device to avoid storing third depth values ​​by discarding third depth values ​​among the first depth values ​​that have a confidence value less than the reference confidence value.

[0214] For example, the one or more programs may include instructions that cause a head-worn electronic device to store depth values ​​based on identifying that the object is different from the reference object. The one or more programs may include instructions that cause a head-worn electronic device to identify another representation corresponding to the object in the second frame image. The one or more programs may include instructions that cause a head-worn electronic device to identify whether the object is located within the second FOV of the one or more sensors using the other representation in the second frame image. The one or more programs may include instructions that cause a head-worn electronic device to generate a depth map for the second frame image that provides depth information for the object located outside the second FOV of the one or more sensors while acquiring the second frame image using the stored depth values ​​based on identifying that the object is located outside the second FOV of the one or more sensors.

[0215] For example, the one or more programs may include instructions that cause a head-worn electronic device to generate the depth map by re-projecting the stored depth values ​​onto the other representation in the second frame image.

[0216] For example, the above one or more programs may include instructions that cause a head-worn electronic device to obtain the depth map providing depth information for the object by using the stored depth values ​​and other depth values ​​obtained through the one or more sensors with respect to the at least part of the object while acquiring the second frame image, based on identifying that at least part of the object is located within the second FOV of the one or more sensors.

[0217] For example, the one or more programs may include instructions that cause a head-worn electronic device to generate the depth map by re-projecting the stored depth values ​​onto the other representation in the second frame image.

[0218] The above-described method may be performed within a head-worn electronic device comprising one or more cameras having a first field of view (FOV) and one or more sensors having a second field of view narrower than the first field of view. The method may include the operation of acquiring a first frame image of the first field of view of the one or more cameras through the one or more cameras. The method may include the operation of identifying an object located within the second field of view of the one or more sensors that corresponds to a representation within the first frame image. The method may include the operation of identifying whether the object corresponds to a reference object. The method may include the operation of storing depth values ​​acquired through the one or more sensors with respect to the object while acquiring the first frame image, in order to generate a depth map for a second frame image to be acquired through the one or more cameras, based on identifying that the object is different from the reference object. The above method may include an operation of refraining from storing depth values ​​by discarding the depth values ​​based on identifying that the object corresponds to the reference object.

[0219] For example, the depth values ​​may be first depth values. The method may include an operation of identifying a distorted region within the representation within the first frame image based on identifying that the object is different from the reference object. The method may include an operation of storing second depth values ​​of the undistorted region within the representation among the first depth values. The method may include an operation of refraining from storing third depth values ​​by discarding third depth values ​​of the distorted region within the representation among the first depth values.

[0220] For example, the depth values ​​may be first depth values. The method may include an operation of identifying confidence values ​​for each of the first depth values ​​based on identifying that the object is different from the reference object. The method may include an operation of storing second depth values ​​among the first depth values ​​that have a confidence value greater than or equal to a reference confidence value. The method may include an operation of refraining from storing third depth values ​​by discarding third depth values ​​among the first depth values ​​that have a confidence value less than the reference confidence value.

[0221] For example, the method may include an operation of storing depth values ​​based on identifying that the object is different from the reference object. The method may include an operation of identifying another representation corresponding to the object in the second frame image. The method may include an operation of identifying whether the object is located within the second FOV of the one or more sensors using the other representation in the second frame image. The method may include an operation of generating a depth map for the second frame image that provides depth information for the object located outside the second FOV of the one or more sensors using the stored depth values ​​while acquiring the second frame image, based on identifying that the object is located outside the second FOV of the one or more sensors.

[0222] For example, the above method may include the operation of generating the depth map by re-projecting the stored depth values ​​onto the other representation in the second frame image.

[0223] For example, the method may include the operation of obtaining the depth map that provides depth information for the object by using the stored depth values ​​and other depth values ​​obtained through the one or more sensors with respect to the at least part of the object while obtaining the second frame image, based on identifying that at least part of the object is located within the second FOV of the one or more sensors.

[0224] For example, the above method may include the operation of generating the depth map by re-projecting the stored depth values ​​onto the other representation in the second frame image.

[0225] The above-described non-transient computer-readable storage medium may store one or more programs. The one or more programs may include instructions that cause the head-wearing electronic device to display a window in a three-dimensional space provided through the display assembly when executed by the head-wearing electronic device comprising one or more cameras having a first field of view (FOV) and one or more sensors having a second FOV narrower than the first FOV. The one or more programs may include instructions that cause the head-wearing electronic device to acquire a first frame image of the first FOV of the one or more cameras through the one or more cameras when executed by the head-wearing electronic device. The one or more programs may include instructions that cause the head-wearing electronic device to identify an object located within the second FOV of the one or more sensors that corresponds to a representation within the first frame image when executed by the head-wearing electronic device. The one or more programs may include instructions that cause the head-wearing electronic device to identify whether the object corresponds to a reference object when executed by the head-wearing electronic device. The one or more programs may include instructions that cause the head-wearing electronic device to store depth values ​​obtained through the one or more sensors with respect to the object while acquiring the first frame image, based on identifying that the object is different from the reference object when executed by the head-wearing electronic device.The above one or more programs may include instructions that cause the head-wearing electronic device to refrain from storing the depth values ​​by discarding the depth values ​​based on identifying that the object corresponds to the reference object when executed by the head-wearing electronic device.

[0226] For example, the depth values ​​may be first depth values. The one or more programs may include instructions that cause the head-wearing electronic device to identify a distorted region within the representation within the first frame image based on identifying that the object is different from the reference object when executed by the head-wearing electronic device. The one or more programs may include instructions that cause the head-wearing electronic device to store second depth values ​​of the undistorted region within the representation among the first depth values ​​when executed by the head-wearing electronic device. The one or more programs may include instructions that cause the head-wearing electronic device to refrain from storing the third depth values ​​by discarding the third depth values ​​of the distorted region within the representation among the first depth values ​​when executed by the head-wearing electronic device.

[0227] For example, the depth values ​​may be first depth values. The one or more programs may include instructions that cause the head-wearing electronic device to identify confidence values ​​of each of the first depth values ​​based on identifying that the object is different from the reference object when executed by the head-wearing electronic device. The one or more programs may include instructions that cause the head-wearing electronic device to store second depth values ​​among the first depth values ​​that have a confidence value greater than or equal to the reference confidence value when executed by the head-wearing electronic device. The one or more programs may include instructions that cause the head-wearing electronic device to refrain from storing third depth values ​​by discarding third depth values ​​among the first depth values ​​that have a confidence value less than the reference confidence value when executed by the head-wearing electronic device.

[0228] For example, the one or more programs may include instructions that cause the head-wearing electronic device to store depth values ​​based on identifying that the object is different from the reference object when executed by the head-wearing electronic device. The one or more programs may include instructions that cause the head-wearing electronic device to identify a different representation corresponding to the object in the second frame image when executed by the head-wearing electronic device. The one or more programs may include instructions that cause the head-wearing electronic device to identify whether the object is located within the second FOV of the one or more sensors using the different representation in the second frame image when executed by the head-wearing electronic device. The above one or more programs may include instructions that cause the head-wearing electronic device to generate a depth map for the second frame image, which provides depth information for the object located outside the second FOV of the one or more sensors while acquiring the second frame image using the stored depth values, based on identifying that the object is located outside the second FOV of the one or more sensors when executed by the head-wearing electronic device.

[0229] For example, the one or more programs may include instructions that cause the head-wearing electronic device to generate the depth map by re-projecting the stored depth values ​​onto the other representation within the second frame image when executed by the head-wearing electronic device.

[0230] For example, the above one or more programs may include instructions that cause the head-wearing electronic device to acquire the depth map providing depth information for the object by using the stored depth values ​​and other depth values ​​acquired through the one or more sensors with respect to the at least part of the object while acquiring the second frame image, based on identifying that at least part of the object is located within the second FOV of the one or more sensors when executed by the head-wearing electronic device.

[0231] For example, the one or more programs may include instructions that cause the head-wearing electronic device to generate the depth map by re-projecting the stored depth values ​​onto the other representation within the second frame image when executed by the head-wearing electronic device.

[0232] The head-wearing electronic device described above may include at least one processor comprising a processing circuit, one or more cameras having a first field of view (FOV), one or more sensors having a second field of view narrower than the first field of view, and a memory comprising one or more storage media for storing one or more programs configured to be executed individually or collectively by the at least one processor. The one or more programs may include instructions that cause the head-wearing electronic device to acquire a first frame image of the first field of view of the one or more cameras through the one or more cameras. The one or more programs may include instructions that cause the head-wearing electronic device to identify an object located within the second field of view of the one or more sensors corresponding to a representation in the first frame image. The above one or more programs may include instructions that cause a head-worn electronic device to identify whether the object is being moved by comparing the first frame image with at least one second frame image prior to the first frame image. The above one or more programs may include instructions that cause a head-worn electronic device to store depth values ​​acquired through the one or more sensors with respect to the object during the acquisition of the first frame image in order to generate a depth map for a third frame image to be acquired through the one or more cameras, based on identifying that the object is not being moved. The above one or more programs may include instructions that cause a head-worn electronic device to refrain from storing the depth values ​​by discarding the depth values, based on identifying that the object is moved.

[0233] For example, the depth values ​​may be first depth values. The one or more programs may include instructions that cause a head-worn electronic device to identify a distorted region within the representation within the first frame image based on identifying that the object has not moved. The one or more programs may include instructions that cause a head-worn electronic device to store second depth values ​​of the undistorted region within the representation among the first depth values. The one or more programs may include instructions that cause a head-worn electronic device to refrain from storing the third depth values ​​by discarding the third depth values ​​of the distorted region within the representation among the first depth values.

[0234] For example, the depth values ​​may be first depth values. The one or more programs may include instructions that cause a head-worn electronic device to identify reliability values ​​for each of the first depth values ​​based on identifying that the object has not moved. The one or more programs may include instructions that cause a head-worn electronic device to store second depth values ​​among the first depth values ​​that have reliability values ​​greater than or equal to a reference reliability value. The one or more programs may include instructions that cause a head-worn electronic device to avoid storing third depth values ​​by discarding third depth values ​​among the first depth values ​​that have reliability values ​​less than the reference reliability value.

[0235] For example, the one or more programs may include instructions that cause a head-worn electronic device to store the depth values ​​based on identifying that the object has not moved. The one or more programs may include instructions that cause a head-worn electronic device to identify another representation corresponding to the object in the third frame image. The one or more programs may include instructions that cause a head-worn electronic device to identify whether the object is located within the second FOV of the one or more sensors using the other representation in the third frame image. The one or more programs may include instructions that cause a head-worn electronic device to generate a depth map for the third frame image that provides depth information for the object located outside the second FOV of the one or more sensors while acquiring the third frame image using the stored depth values ​​based on identifying that the object is located outside the second FOV of the one or more sensors.

[0236] For example, the above one or more programs may include instructions that cause a head-worn electronic device to generate the depth map by re-projecting the stored depth values ​​onto the other representation in the third frame image.

[0237] For example, the above one or more programs may include instructions that cause a head-worn electronic device to acquire the depth map providing depth information for the object by using the stored depth values ​​and other depth values ​​acquired through the one or more sensors with respect to the at least part of the object while acquiring the third frame image, based on identifying that at least part of the object is located within the second FOV of the one or more sensors.

[0238] For example, the above one or more programs may include instructions that cause a head-worn electronic device to generate the depth map by re-projecting the stored depth values ​​onto the other representation in the third frame image.

[0239] The above-described method may be performed within a head-worn electronic device comprising one or more cameras having a first field of view (FOV) and one or more sensors having a second FOV narrower than the first FOV. The method may include the operation of acquiring a first frame image of the first FOV of the one or more cameras through the one or more cameras. The method may include the operation of identifying an object located within the second FOV of the one or more sensors that corresponds to a representation within the first frame image. The method may include the operation of identifying whether the object is being moved by comparing the first frame image with at least one second frame image prior to the first frame image. The method may include the operation of storing depth values ​​acquired through the one or more sensors with respect to the object while acquiring the first frame image, in order to generate a depth map for a third frame image to be acquired through the one or more cameras, based on identifying that the object is not being moved. The above method may include an operation of refraining from storing depth values ​​by discarding the depth values ​​based on identifying that the object has been moved.

[0240] For example, the depth values ​​may be first depth values. The method may include an operation of identifying a distorted region within the representation within the first frame image based on identifying that the object has not moved. The method may include an operation of storing second depth values ​​of the undistorted region within the representation among the first depth values. The method may include an operation of refraining from storing third depth values ​​by discarding third depth values ​​of the distorted region within the representation among the first depth values.

[0241] For example, the depth values ​​may be first depth values. The method may include an operation of identifying reliability values ​​for each of the first depth values ​​based on identifying that the object has not moved. The method may include an operation of storing second depth values ​​among the first depth values ​​that have reliability values ​​greater than or equal to a reference reliability value. The method may include an operation of refraining from storing third depth values ​​by discarding third depth values ​​among the first depth values ​​that have reliability values ​​less than the reference reliability value.

[0242] For example, the method may include an operation of storing depth values ​​based on identifying that the object has not moved. The method may include an operation of identifying another representation corresponding to the object in the third frame image. The method may include an operation of identifying whether the object is located within the second FOV of the one or more sensors using the other representation in the third frame image. The method may include an operation of generating a depth map for the third frame image that provides depth information for the object located outside the second FOV of the one or more sensors while acquiring the third frame image, using the stored depth values ​​based on identifying that the object is located outside the second FOV of the one or more sensors.

[0243] For example, the above method may include the operation of generating the depth map by re-projecting the stored depth values ​​onto the other representation in the third frame image.

[0244] For example, the method may include the operation of obtaining the depth map that provides depth information about the object by using the stored depth values ​​and other depth values ​​obtained through the one or more sensors with respect to the at least part of the object while obtaining the third frame image, based on identifying that at least part of the object is located within the second FOV of the one or more sensors.

[0245] For example, the above method may include the operation of generating the depth map by re-projecting the stored depth values ​​onto the other representation in the third frame image.

[0246] The above-described non-transient computer-readable storage medium may store one or more programs. The one or more programs may include instructions that cause the head-wearing electronic device to display a window in a three-dimensional space provided through the display assembly when executed by the head-wearing electronic device comprising one or more cameras having a first field of view (FOV) and one or more sensors having a second FOV narrower than the first FOV. The one or more programs may include instructions that cause the head-wearing electronic device to acquire a first frame image of the first FOV of the one or more cameras through the one or more cameras when executed by the head-wearing electronic device. The one or more programs may include instructions that cause the head-wearing electronic device to identify an object located within the second FOV of the one or more sensors that corresponds to a representation within the first frame image when executed by the head-wearing electronic device. The above one or more programs may include instructions that cause the head-wearing electronic device to identify whether the object is being moved by comparing the first frame image with at least one second frame image prior to the first frame image when executed by the head-wearing electronic device.The above one or more programs may include instructions that cause the head-wearing electronic device to store depth values ​​acquired through the one or more sensors with respect to the object during the acquisition of the first frame image to generate a depth map for a third frame image to be acquired through the one or more cameras, based on identifying that the object is not being moved when executed by the head-wearing electronic device. The above one or more programs may include instructions that cause the head-wearing electronic device to refrain from storing the depth values ​​by discarding the depth values, based on identifying that the object has been moved when executed by the head-wearing electronic device.

[0247] The depth values ​​may be first depth values. The one or more programs may include instructions that cause the head-wearing electronic device to identify a distorted region within the representation within the first frame image based on identifying that the object has not moved when executed by the head-wearing electronic device. The one or more programs may include instructions that cause the head-wearing electronic device to store second depth values ​​of the undistorted region within the representation among the first depth values ​​when executed by the head-wearing electronic device. The one or more programs may include instructions that cause the head-wearing electronic device to refrain from storing the third depth values ​​by discarding the third depth values ​​of the distorted region within the representation among the first depth values ​​when executed by the head-wearing electronic device.

[0248] The depth values ​​may be first depth values. The one or more programs may include instructions that cause the head-wearing electronic device to identify reliability values ​​for each of the first depth values ​​based on identifying that the object has not moved when executed by the head-wearing electronic device. The one or more programs may include instructions that cause the head-wearing electronic device to store second depth values ​​among the first depth values ​​that have reliability values ​​greater than or equal to a reference reliability value when executed by the head-wearing electronic device. The one or more programs may include instructions that cause the head-wearing electronic device to refrain from storing third depth values ​​by discarding third depth values ​​among the first depth values ​​that have reliability values ​​less than the reference reliability value when executed by the head-wearing electronic device.

[0249] For example, the one or more programs may include instructions that cause the head-wearing electronic device to store depth values ​​based on identifying that the object has not moved when executed by the head-wearing electronic device. The one or more programs may include instructions that cause the head-wearing electronic device to identify another representation corresponding to the object in the third frame image when executed by the head-wearing electronic device. The one or more programs may include instructions that cause the head-wearing electronic device to identify whether the object is located within the second FOV of the one or more sensors using the other representation in the third frame image when executed by the head-wearing electronic device. The above one or more programs may include instructions that cause the head-wearing electronic device to generate a depth map for the third frame image, which provides depth information for the object located outside the second FOV of the one or more sensors while acquiring the third frame image using the stored depth values, based on identifying that the object is located outside the second FOV of the one or more sensors when executed by the head-wearing electronic device.

[0250] For example, the one or more programs may include instructions that cause the head-wearing electronic device to generate the depth map by re-projecting the stored depth values ​​onto the other representation within the third frame image when executed by the head-wearing electronic device.

[0251] For example, the above one or more programs may include instructions that cause the head-wearing electronic device to acquire the depth map providing depth information for the object by using the stored depth values ​​and other depth values ​​acquired through the one or more sensors with respect to the at least part of the object while acquiring the third frame image, based on identifying that at least part of the object is located within the second FOV of the one or more sensors when executed by the head-wearing electronic device.

[0252] For example, the one or more programs may include instructions that cause the head-wearing electronic device to generate the depth map by re-projecting the stored depth values ​​onto the other representation within the third frame image when executed by the head-wearing electronic device.

[0253] The head-wearing electronic device described above may include at least one processor comprising a processing circuit, one or more cameras having a first field of view (FOV), one or more sensors having a second field of view narrower than the first field of view, and a memory comprising one or more storage media configured to store one or more programs configured to be executed individually or collectively by the at least one processor. The one or more programs may include instructions that cause the head-wearing electronic device to acquire a first frame image of the first field of view of the one or more cameras, which includes a representation of an object, through the one or more cameras. The object may be located within the first field of view of the one or more cameras and the second field of view of the one or more sensors while acquiring the first frame image. The one or more programs may include instructions that cause the head-wearing electronic device to acquire depth values ​​of the object through the one or more sensors and store said depth values ​​while acquiring the first frame image. The above one or more programs may include instructions that cause a head-worn electronic device to acquire a second frame image of the first FOV of the one or more cameras, including a different representation of the object, through the one or more cameras after acquiring the first frame image. The object may be located within the first FOV of the one or more cameras and outside the second FOV of the one or more sensors while acquiring the second frame image.The above one or more programs may include instructions that cause a head-worn electronic device to generate a depth map for the second frame image, which provides depth information for the object located outside the second FOV of the one or more sensors while acquiring the second frame image, using the depth values.

[0254] For example, the one or more programs may include instructions that cause a head-worn electronic device to acquire the depth values ​​of the object through the one or more sensors while acquiring the first frame image. The one or more programs may include instructions that cause a head-worn electronic device to identify whether the object corresponds to a reference object. The one or more programs may include instructions that cause a head-worn electronic device to store the depth values ​​based on identifying that the object is different from the reference object.

[0255] For example, the one or more programs may include instructions that cause a head-worn electronic device to acquire the depth values ​​of the object through the one or more sensors while acquiring the first frame image. The one or more programs may include instructions that cause a head-worn electronic device to identify whether the object is being moved by comparing the first frame image with at least one third frame image prior to the first frame image. The one or more programs may include instructions that cause a head-worn electronic device to store the depth values ​​based on identifying that the object is not being moved.

[0256] For example, the depth values ​​may be first depth values. The one or more programs may include instructions that cause a head-worn electronic device to acquire the first depth values ​​of the object through the one or more sensors while acquiring the first frame image. The one or more programs may include instructions that cause a head-worn electronic device to identify a distorted region within the representation within the first frame image. The one or more programs may include instructions that cause a head-worn electronic device to store second depth values ​​of the undistorted region within the representation among the first depth values. The one or more programs may include instructions that cause a head-worn electronic device to refrain from storing the third depth values ​​by discarding the third depth values ​​of the distorted region within the representation among the first depth values.

[0257] For example, the depth values ​​may be first depth values. The one or more programs may include instructions that cause a head-worn electronic device to acquire the first depth values ​​of the object through the one or more sensors while acquiring the first frame image. The one or more programs may include instructions that cause a head-worn electronic device to identify the reliability values ​​of each of the first depth values. The one or more programs may include instructions that cause a head-worn electronic device to store second depth values ​​among the first depth values ​​that have a reliability value greater than or equal to a reference reliability value. The one or more programs may include instructions that cause a head-worn electronic device to refrain from storing third depth values ​​by discarding third depth values ​​among the first depth values ​​that have a reliability value less than the reference reliability value.

[0258] For example, the one or more programs may include instructions that cause a head-worn electronic device to generate the depth map by re-projecting the stored depth values ​​onto the other representation in the second frame image.

[0259] The above-described method may be performed within a head-worn electronic device comprising one or more cameras having a first field of view (FOV) and one or more sensors having a second field of view narrower than the first field of view. The method may include the operation of acquiring a first frame image of the first field of view of the one or more cameras through the one or more cameras. The method may include the operation of acquiring a first frame image of the first field of view of the one or more cameras, including a representation of an object, through the one or more cameras. The object may be located within the first field of view of the one or more cameras and the second field of view of the one or more sensors while acquiring the first frame image. The method may include the operation of acquiring depth values ​​of the object through the one or more sensors and storing the depth values ​​while acquiring the first frame image. The above method may include the operation of acquiring a second frame image of the first FOV of the one or more cameras, including a different representation of the object, through the one or more cameras after acquiring the first frame image. The object may be located within the first FOV of the one or more cameras and outside the second FOV of the one or more sensors while acquiring the second frame image. The above method may include the operation of generating a depth map for the second frame image that provides depth information for the object located outside the second FOV of the one or more sensors while acquiring the second frame image, using the depth values.

[0260] For example, the method may include an operation of acquiring depth values ​​of the object through one or more sensors while acquiring the first frame image. The method may include an operation of identifying whether the object corresponds to a reference object. The method may include an operation of storing the depth values ​​based on identifying that the object is different from the reference object.

[0261] For example, the method may include the operation of acquiring depth values ​​of the object through one or more sensors while acquiring the first frame image. The method may include the operation of identifying whether the object is being moved by comparing the first frame image with at least one third frame image prior to the first frame image. The method may include the operation of storing the depth values ​​based on identifying that the object is not being moved.

[0262] For example, the depth values ​​may be first depth values. The method may include the operation of acquiring the first depth values ​​of the object through one or more sensors while acquiring the first frame image. The method may include the operation of identifying a distorted region within the representation within the first frame image. The method may include the operation of storing second depth values ​​of the undistorted region within the representation among the first depth values. The method may include the operation of refraining from storing the third depth values ​​by discarding the third depth values ​​of the distorted region within the representation among the first depth values.

[0263] For example, the depth values ​​may be first depth values. The method may include the operation of acquiring the first depth values ​​of the object through one or more sensors while acquiring the first frame image. The method may include the operation of identifying reliability values ​​for each of the first depth values. The method may include the operation of storing second depth values ​​among the first depth values ​​that have reliability values ​​greater than or equal to a reference reliability value. The method may include the operation of refraining from storing third depth values ​​by discarding third depth values ​​among the first depth values ​​that have reliability values ​​less than the reference reliability value.

[0264] For example, the above method may include the operation of generating the depth map by re-projecting the stored depth values ​​onto the other representation in the second frame image.

[0265] The above-described non-transient computer-readable storage medium may store one or more programs. The one or more programs may include instructions that cause the head-wearing electronic device to acquire a first frame image of the first FOV of the one or more cameras, which includes a representation of an object, through the one or more cameras, when executed by the head-wearing electronic device comprising one or more cameras having a first FOV (field of view) and one or more sensors having a second FOV narrower than the first FOV. The object may be located within the first FOV of the one or more cameras and the second FOV of the one or more sensors while acquiring the first frame image. The one or more programs may include instructions that cause the head-wearing electronic device to acquire depth values ​​of the object through the one or more sensors and store said depth values ​​while acquiring the first frame image, when executed by the head-wearing electronic device. The above one or more programs may include instructions that cause the head-wearing electronic device, when executed by the head-wearing electronic device, to acquire the first frame image and then, through the one or more cameras, acquire a second frame image of the first FOV of the one or more cameras containing a different representation of the object. The object may be located within the first FOV of the one or more cameras and outside the second FOV of the one or more sensors while acquiring the second frame image.The above one or more programs may include instructions that cause the head-wearing electronic device to generate a depth map for the second frame image, which provides depth information for the object located outside the second FOV of the one or more sensors while acquiring the second frame image, using the depth values ​​when executed by the head-wearing electronic device.

[0266] For example, the one or more programs may include instructions that cause the head-wearing electronic device to acquire the depth values ​​of the object through the one or more sensors while acquiring the first frame image when executed by the head-wearing electronic device. The one or more programs may include instructions that cause the head-wearing electronic device to identify whether the object corresponds to a reference object when executed by the head-wearing electronic device. The one or more programs may include instructions that cause the head-wearing electronic device to store the depth values ​​based on identifying that the object is different from the reference object when executed by the head-wearing electronic device.

[0267] For example, the one or more programs may include instructions that cause the head-wearing electronic device to acquire the depth values ​​of the object through the one or more sensors while acquiring the first frame image when executed by the head-wearing electronic device. The one or more programs may include instructions that cause the head-wearing electronic device to identify whether the object is being moved by comparing the first frame image with at least one third frame image prior to the first frame image when executed by the head-wearing electronic device. The one or more programs may include instructions that cause the head-wearing electronic device to store the depth values ​​based on identifying that the object is not being moved when executed by the head-wearing electronic device.

[0268] For example, the depth values ​​may be first depth values. The one or more programs may include instructions that cause the head-wearing electronic device to acquire the first depth values ​​of the object through the one or more sensors while acquiring the first frame image when executed by the head-wearing electronic device. The one or more programs may include instructions that cause the head-wearing electronic device to identify a distorted region within the representation within the first frame image when executed by the head-wearing electronic device. The one or more programs may include instructions that cause the head-wearing electronic device to store second depth values ​​of the undistorted region within the representation among the first depth values ​​when executed by the head-wearing electronic device. The above one or more programs may include instructions that cause the head-wearing electronic device to refrain from storing the third depth values ​​by discarding the third depth values ​​of the distorted region within the representation among the first depth values ​​when executed by the head-wearing electronic device.

[0269] For example, the depth values ​​may be first depth values. The one or more programs may include instructions that cause the head-wearing electronic device to acquire the first depth values ​​of the object through the one or more sensors while acquiring the first frame image when executed by the head-wearing electronic device. The one or more programs may include instructions that cause the head-wearing electronic device to identify the reliability values ​​of each of the first depth values ​​when executed by the head-wearing electronic device. The one or more programs may include instructions that cause the head-wearing electronic device to store second depth values ​​among the first depth values ​​that have a reliability value greater than or equal to a reference reliability value when executed by the head-wearing electronic device. The above one or more programs may include instructions that cause the head-wearing electronic device to refrain from storing the third depth values ​​by discarding the third depth values ​​among the first depth values ​​that have a reliability value less than the reference reliability value when executed by the head-wearing electronic device.

[0270] For example, the one or more programs may include instructions that cause the head-wearing electronic device to generate the depth map by re-projecting the stored depth values ​​onto the other representation within the second frame image when executed by the head-wearing electronic device.

[0271] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs.

Claims

1. In a head-worn electronic device, At least one processor including a processing circuit; One or more cameras having a first FOV (field of view); One or more sensors having a second FOV narrower than the first FOV; and Memory comprising one or more programs configured to be executed individually or collectively by at least one processor, and including one or more storage media, The above one or more programs are, A first frame image of the first FOV of the one or more cameras is obtained through the one or more cameras, and Identifying an object located within the second FOV of the one or more sensors that corresponds to a representation within the first frame image, and Identify whether the above object corresponds to a reference object, and Based on identifying that the object is different from the reference object, depth values ​​obtained through the one or more sensors with respect to the object while acquiring the first frame image are stored to generate a depth map for a second frame image to be acquired through the one or more cameras, and Based on identifying that the above object corresponds to the above reference object, the depth values ​​are discarded to avoid storing the depth values. Instructions including those that cause the above-mentioned head-worn electronic device Head-worn electronic device.

2. In Claim 1, The above depth values ​​are, These are the first depth values, The above one or more programs are, Based on identifying that the above object is different from the above reference object, a distorted region within the representation within the first frame image is identified, and Among the first depth values ​​above, the second depth values ​​of the undistorted region within the representation are stored, and To avoid storing the third depth values ​​by discarding the third depth values ​​of the distorted region within the expression among the first depth values, Instructions including those that cause the above-mentioned head-worn electronic device Head-worn electronic device.

3. In Claim 1, The above depth values ​​are, These are the first depth values, The above one or more programs are, Based on identifying that the above object is different from the above reference object, the confidence values ​​of each of the above first depth values ​​are identified, and Among the first depth values ​​above, second depth values ​​having a reliability value greater than or equal to a reference reliability value are stored, and By discarding the third depth values ​​among the first depth values ​​that have a reliability value less than the reference reliability value, the storage of the third depth values ​​is avoided. Instructions including those that cause the above-mentioned head-worn electronic device Head-worn electronic device.

4. In Claim 1, The above one or more programs are, Based on identifying that the above object is different from the above reference object, the depth values ​​are stored, and Identifying another representation corresponding to the object within the second frame image, and Using the other representation within the second frame image, identifying whether the object is located within the second FOV of the one or more sensors, and Based on identifying that the object is located outside the second FOV of the one or more sensors, to generate the depth map for the second frame image that provides depth information for the object located outside the second FOV of the one or more sensors while acquiring the second frame image using the stored depth values. Instructions including those that cause the above-mentioned head-worn electronic device Head-worn electronic device.

5. In Claim 4, The above one or more programs are, To generate the depth map by re-projecting the stored depth values ​​onto the other representation within the second frame image, Instructions including those that cause the above-mentioned head-worn electronic device Head-worn electronic device.

6. In Claim 4, The above one or more programs are, Based on identifying that at least a portion of the object is located within the second FOV of the one or more sensors, the depth map providing depth information for the object is generated using the stored depth values ​​and other depth values ​​obtained through the one or more sensors with respect to the at least portion of the object while acquiring the second frame image. Instructions including those that cause the above-mentioned head-worn electronic device Head-worn electronic device.

7. In Claim 6, The above one or more programs are, To generate the depth map by re-projecting the stored depth values ​​onto the other representation within the second frame image, Instructions including those that cause the above-mentioned head-worn electronic device Head-worn electronic device.

8. In a head-worn electronic device, At least one processor including a processing circuit; One or more cameras having a first FOV (field of view); One or more sensors having a second FOV narrower than the first FOV; and Memory comprising one or more programs configured to be executed individually or collectively by at least one processor, and including one or more storage media, The above one or more programs are, A first frame image of the first FOV of the one or more cameras is obtained through the one or more cameras, and Identifying an object located within the second FOV of the one or more sensors that corresponds to a representation within the first frame image, and Identify whether the object is being moved by comparing the first frame image with at least one second frame image prior to the first frame image, and Based on identifying that the object is not being moved, depth values ​​obtained through the one or more sensors with respect to the object while acquiring the first frame image are stored to generate a depth map for a third frame image to be acquired through the one or more cameras, and Based on identifying that the above object has been moved, the depth values ​​are discarded to avoid storing the depth values. Instructions including those that cause the above-mentioned head-worn electronic device Head-worn electronic device.

9. In Claim 8, The above depth values ​​are, These are the first depth values, The above one or more programs are, Based on identifying that the above object has not moved, a distorted region within the representation within the first frame image is identified, and Among the first depth values ​​above, the second depth values ​​of the undistorted region within the representation are stored, and To avoid storing the third depth values ​​by discarding the third depth values ​​of the distorted region within the expression among the first depth values, Instructions including those that cause the above-mentioned head-worn electronic device Head-worn electronic device.

10. In Claim 8, The above depth values ​​are, These are the first depth values, The above one or more programs are, Based on identifying that the above object has not moved, the reliability values ​​of each of the first depth values ​​are identified, and Among the first depth values ​​above, second depth values ​​having a reliability value greater than or equal to a reference reliability value are stored, and By discarding the third depth values ​​among the first depth values ​​that have a reliability value less than the reference reliability value, the storage of the third depth values ​​is avoided. Instructions including those that cause the above-mentioned head-worn electronic device Head-worn electronic device.

11. In Claim 8, The above one or more programs are, Based on identifying that the above object has not moved, the depth values ​​are stored, and Identifying other representations corresponding to the object within the third frame image, and Using the other representation within the third frame image, identifying whether the object is located within the second FOV of the one or more sensors, and Based on identifying that the object is located outside the second FOV of the one or more sensors, to generate the depth map for the third frame image that provides depth information for the object located outside the second FOV of the one or more sensors while acquiring the third frame image using the stored depth values. Instructions including those that cause the above-mentioned head-worn electronic device Head-worn electronic device.

12. In Claim 11, The above one or more programs are, To generate the depth map by re-projecting the stored depth values ​​onto the other representation within the third frame image, Instructions including those that cause the above-mentioned head-worn electronic device Head-worn electronic device.

13. In Claim 11, The above one or more programs are, Based on identifying that at least a portion of the object is located within the second FOV of the one or more sensors, the depth map providing depth information for the object is generated using the stored depth values ​​and other depth values ​​obtained through the one or more sensors with respect to the at least portion of the object while acquiring the third frame image. Instructions including those that cause the above-mentioned head-worn electronic device Head-worn electronic device.

14. In Claim 13, The above one or more programs are, To generate the depth map by re-projecting the stored depth values ​​onto the other representation within the third frame image, Instructions including those that cause the above-mentioned head-worn electronic device Head-worn electronic device.

15. In a head-worn electronic device, At least one processor including a processing circuit; One or more cameras having a first FOV (field of view); One or more sensors having a second FOV narrower than the first FOV; and Memory comprising one or more programs configured to be executed individually or collectively by at least one processor, and including one or more storage media, The above one or more programs are: Acquiring a first frame image of the first FOV of the one or more cameras including a representation of an object through the one or more cameras; and the object is located within the first FOV of the one or more cameras and the second FOV of the one or more sensors while acquiring the first frame image. While acquiring the first frame image, depth values ​​of the object are acquired through the one or more sensors, and the depth values ​​are stored; After acquiring the first frame image, a second frame image of the first FOV of the one or more cameras, including a different representation of the object, is acquired through the one or more cameras; and while acquiring the second frame image, the object is located within the first FOV of the one or more cameras and outside the second FOV of the one or more sensors. Using the depth values ​​above, to generate a depth map for the second frame image that provides depth information for the object located outside the second FOV of the one or more sensors while acquiring the second frame image, Instructions including those that cause the above-mentioned head-worn electronic device Head-worn electronic device.

Images