Electronic device and method for displaying image

Abstract

The present disclosure provides an electronic device and an operating method of the electronic device. The electronic device may comprise: a display including a display panel and at least one optical element; a depth sensor; a memory including at least one storage medium for storing instructions; and at least one processor. The at least one processor acquires, by means of the depth sensor, a first depth map of a space including a reference object, wherein the first depth map can include depth information for each pixel. The at least one processor can adjust, on the basis of the first depth map, an arrangement of the display so that an image is formed at a position in a space related to the reference object, and the arrangement adjustment of the display can include a change in the position or direction of the display panel and / or the at least one optical element. The at least one processor can display an image on the display.

Description

Electronic device and method for displaying an image

[0001] The present disclosure relates to an electronic device for displaying an image and a method of operating the electronic device.

[0002] Display devices are widely used to visually provide users with various data, such as text and images (including videos). Recently, so-called floating display technology, which makes images appear to float in the air without a physical screen, has been gaining attention. Examples of floating display implementations include technologies such as AIRR (aerial imaging by retro-reflection) and holographic displays.

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0004] According to one embodiment of the present disclosure, an electronic device may be provided. The electronic device may include a display, a depth sensor, a camera, a motor subsystem, a memory comprising at least one storage medium storing at least one instruction, and at least one processor capable of executing said at least one instruction. The at least one processor may control the operation method of the electronic device by executing at least one instruction.

[0005] According to one embodiment, at least one processor acquires a first depth map of a space containing a reference object through a depth sensor, the first depth map includes pixel-wise depth information, and based on the first depth map, adjusts the arrangement of a display so that an image is formed at a position in space related to the reference object, the arrangement of the display includes changing the position or orientation of at least one of a display panel or at least one optical element included in the display, and can display an image through the display.

[0006] According to one embodiment, the reference object may include a hand that performs a defined gesture.

[0007] According to one embodiment, at least one processor can identify the locations of a plurality of joint points of a hand performing a gesture based on a first depth map, determine a display area in space based on the locations of the plurality of joint points, and adjust the arrangement of the display so that an image is formed at the location of the display area.

[0008] According to one embodiment, at least one processor obtains a normal vector of at least one plane based on the positions of a plurality of joint points, and the at least one plane includes a plane through which three joint points included in the plurality of joint points pass, and can determine the position and orientation of a display area based on at least one normal vector for the at least one plane and the positions of the plurality of joint points. The display area may be a planar area.

[0009] According to one embodiment, at least one processor can obtain a covariance matrix based on the positions of a plurality of joint points, obtain eigenvalues ​​of the covariance matrix and eigenvectors corresponding to the eigenvalues, and determine the position and orientation of the display area based on the eigenvector corresponding to the smallest eigenvalue among the eigenvalues ​​and the positions of the plurality of joint points.

[0010] According to one embodiment, at least one processor may adjust the arrangement of the display to adjust the size of the image to be formed as the image is displayed. The size of the image to be formed may be related to at least one of the depth at which the image is formed or the projection angle of the display.

[0011] According to one embodiment, at least one processor identifies whether the user is a pre-registered user based on identifying the user, provides guidance on performing a defined gesture based on identifying that the user is a pre-registered user, stores identification information of the user, determines a display area in space based on the position of the plurality of joint points of the hand performing the gesture, and can adjust the arrangement of the display so that an image is formed in the display area.

[0012] According to one embodiment, at least one processor can identify whether the user performs the gesture based on identifying that the user is a pre-registered user. If the user performs the gesture, the at least one processor can determine a display area in space based on the positions of the plurality of joint points of the hand performing the gesture and adjust the arrangement of the display so that an image is formed in the display area. If the user does not perform the gesture, the at least one processor can adjust the arrangement of the display based on pre-stored setting information mapped to the user's identification information.

[0013] According to one embodiment, at least one processor can identify whether the user is a pre-registered user based on data regarding the user's face.

[0014] According to one embodiment, at least one processor identifies user access and identifies the user's body information, said body information includes the user's height, and based on the user's body information, can adjust the size of the image to be formed as the image is displayed.

[0015] According to one embodiment, at least one processor acquires a second depth map of an image formed by displaying an image and a third depth map of a touch input object, identifies whether the positional relationship between the image and the touch input object satisfies at least one condition based on the second depth map and the third depth map, and can recognize a touch input based on identifying that the at least one condition is satisfied. The at least one condition may include at least one of a condition in which the touch input object contacts the image, a condition in which the distance of the touch input object from the image is less than or equal to a first threshold, a condition in which the depth through which the touch input object penetrates the image is greater than or equal to a second threshold, or a condition in which at least one of the above conditions is continuously satisfied for a threshold time or longer.

[0016] According to one embodiment of the present disclosure, a method of operating an electronic device may be provided. The method of operating an electronic device may include at least one operation. The at least one operation may include at least one of: an operation of acquiring a first depth map of a space containing a reference object; an operation of adjusting the arrangement of a display of the electronic device so that an image is formed at a position in space related to the reference object based on the first depth map; and an operation of displaying an image through the display.

[0017] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0018] FIG. 1 illustrates a block diagram of an electronic device according to one embodiment of the present disclosure.

[0019] FIG. 2 is a drawing for explaining the process of forming an image through a display according to one embodiment of the present disclosure.

[0020] FIGS. 3a and 3b are drawings for explaining the operation of an electronic device adjusting a position according to one embodiment of the present disclosure.

[0021] FIG. 4a is a drawing for explaining the relationship between the depth at which a phase is formed and the size of the phase according to one embodiment of the present disclosure.

[0022] FIG. 4b is a diagram illustrating the relationship between the projection angle and the size of an image of a display according to one embodiment of the present disclosure.

[0023] FIG. 5 is a drawing for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0024] FIG. 6 illustrates the joint points and skeleton of a hand according to one embodiment of the present disclosure.

[0025] FIG. 7 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure determining a display area.

[0026] FIG. 8 is a drawing for explaining the size adjustment of a display area according to one embodiment of the present disclosure.

[0027] FIG. 9 is a drawing for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0028] FIG. 10 is a drawing for explaining touch input according to one embodiment of the present disclosure.

[0029] FIG. 11 illustrates the configuration of a motor subsystem according to one embodiment of the present disclosure.

[0030] FIG. 12a illustrates actuators according to one embodiment of the present disclosure.

[0031] FIG. 12b illustrates an actuator coupled with a ball joint according to one embodiment of the present disclosure.

[0032] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily implement them. However, the present disclosure may be embodied in various different forms and should be understood to include various modifications, equivalents, or substitutions of the embodiments described herein, rather than being limited to the embodiments described herein. The present disclosure is capable of various modifications by those skilled in the art without departing from the gist of the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

[0033] The purposes and effects of the present disclosure are not limited to those mentioned in the drawings and the following related description, and various modifications may be made within the technical scope of the present disclosure. The effects according to the embodiments of the present disclosure mentioned below are merely illustrative and are not limited thereto; depending on various modifications, different or additional effects may be realized.

[0034] In the following drawings and related descriptions, functions, configurations, technical terms, and technical details well known in the art to which this disclosure pertains may be omitted. This is intended to convey the essentials of this disclosure more clearly and concisely by minimizing unnecessary detailed descriptions.

[0035] In the drawings, each block of the flowcharts and combinations of the flowcharts may be performed by at least one instruction. The instruction may be loaded into a processor of a computer or other programmable data processing equipment to generate means for performing the functions described in the drawings. The instruction may also provide steps for performing the functions described in the drawings by being executed on a computer or other programmable data processing equipment.

[0036] Meanwhile, various elements and regions in the drawings are depicted schematically, and the technical concept of the present disclosure is not limited by the relative sizes, spacing, or arrangements depicted in the attached drawings. The electronic device of the present disclosure is not limited to the configuration and / or operation shown in the drawings and may include all other configurations capable of performing the same or similar functions.

[0037] The individual components depicted in the drawings are not required to be implemented in a physically separate form, but are shown separately to aid in the description and understanding of the present disclosure. The present disclosure may be implemented in a form in which the individual components shown in the drawings are merged, modified, or have some components deleted and / or added. Each component may perform functions in conjunction with one another while existing in physically separated locations via a network or communication link.

[0038] Likewise, the operations depicted in the drawings are illustrative to aid in the description and understanding of the present disclosure, and the present disclosure may be modified by merging, changing the order of, or deleting and / or adding parts of the operations shown in the drawings. For example, two or more operations shown consecutively in the drawings may be performed simultaneously, in reverse order as necessary, repeatedly, or omitted depending on the actual situation.

[0039] FIG. 1 illustrates a block configuration of an electronic device according to one embodiment of the present disclosure.

[0040] The electronic device (100) of FIG. 1 may be a smartphone, tablet PC, PC, smart TV, mobile phone, PDA (personal digital assistant), laptop, media player, micro server, digital broadcasting terminal, navigation, kiosk, home appliance, and other mobile or non-mobile computing devices, but is not limited thereto. Additionally, the electronic device (100) may perform various computing functions such as real-time video viewing and communication. The embodiments of the present disclosure regarding the electronic device (100) described below may be equally applicable to other electronic devices capable of displaying images.

[0041] According to one embodiment, the electronic device (100) may include at least one processor (110), memory (120), display (130), depth sensor (140), camera (150) and / or motor subsystem (160).

[0042] According to one embodiment, the memory (120) is a storage medium used by the electronic device (100) and can store data such as at least one instruction (121) or setting information corresponding to at least one program. The program may include an operating system (OS) program and various application programs. When the at least one instruction (121) stored in the memory (120) is executed by at least one processor (110), it can cause the electronic device (100) to perform at least one operation.

[0043] According to one embodiment, the memory (120) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), RAM (random access memory, RAM), SRAM (static random access memory), ROM (read only memory, ROM), EEPROM (electrically erasable programmable ROM), PROM (programmable ROM), magnetic memory, a magnetic disk, and an optical disk.

[0044] According to one embodiment, the display (130) can perform functions for outputting information in the form of numbers, characters, images, and / or graphics, and can display a screen corresponding to data received from at least one processor (110). The display (130) may include, for example, a control circuit for controlling the display (130).

[0045] According to one embodiment, the display (130) may include a display panel (131) and at least one optical element (132). The display (130) may be a floating display that forms an image in the air rather than a physical screen.

[0046] According to one embodiment, the display panel (131) may include at least one hardware module for visual output. The at least one hardware module may include, for example, at least one of an LCD (Liquid Crystal Display), an LED (Light Emitting Diode), an LPD (Light Emitting Polymer Display), an OLED (Organic Light Emitting Diode), an AMOLED (Active Matrix Organic Light Emitting Diode), or an FLED (Flexible LED).

[0047] According to one embodiment, the display panel (131) may include a polarizing filter. If the display panel (131) includes a polarizing filter, the display panel (131) may selectively emit light of a specific polarization state.

[0048] According to one embodiment, at least one optical element (132) may include an optical hardware configuration for forming an image at a desired location in the air from an image generated on a display panel (131). For example, at least one optical element (132) may include various optical elements related to controlling an optical path, such as a lens, a reflector, a diffraction optical element (DOE), a prism, a waveguide, a beam splitter, and an optical diffuser.

[0049] According to one embodiment, at least one optical element (132) may include a retroreflector and a beam splitter. The retroreflector may be an optical element that reflects incident light back in the opposite direction. The retroreflector may be implemented, for example, as a corner cube prism structure or a microsphere bead structure. The beam splitter may be an optical element that transmits some of the incident light and reflects others. The beam splitter may be implemented in various forms, such as a prism-based beam splitter, a polarizing beam splitter, or an optically coated plate-based beam splitter.

[0050] According to one embodiment, the display panel (131) may be referred to as a 'light source' or 'panel', and the display (130) may be referred to as an 'output unit', 'display unit', or other terms having an equivalent technical meaning.

[0051] According to one embodiment, the depth sensor (140) may include a sensor capable of measuring the distance or depth between the electronic device (100) and the external environment. The depth sensor (140) may operate using various sensing technologies, such as, for example, infrared (IR), laser-based technology (LIDAR), structured light, or ultrasonic technology.

[0052] According to one embodiment, the depth sensor (140) may include a time-of-flight (ToF) sensor. The ToF sensor may be a sensor that measures distance or depth by measuring the time it takes for light emitted from a light source to be reflected back from an object. The ToF sensor may be used to obtain high-resolution depth information, for example, by providing high precision at the level of several millimeters.

[0053] According to one embodiment, the camera (150) may include components capable of capturing images or recording video. The camera (150) may include various types of imaging sensors, such as RGB (red, green, blue) sensors, IR sensors, and / or thermal sensors. The camera (150) may be used in applications such as object recognition, face recognition, motion tracking, and augmented reality (AR). Additionally, the camera (150) may capture visual information of the surrounding environment and store it in a display (130) or memory (120), or transmit it to another device.

[0054] According to one embodiment, the depth sensor (140) and the camera (150) may be implemented as a single integrated component. For example, the depth sensor (140) and the camera (150) may be implemented as a single hardware module (e.g., ToF camera, RGB-D camera) that generates depth information and 2D pixel information (e.g., pixel-by-pixel RGB information) at once. For example, the camera (150) may be implemented as a configuration included in the depth sensor (140). In the present disclosure, data obtained through the depth sensor (140) and / or the camera (150) may be expressed as data obtained through either the depth sensor (140) or the camera (150), since the depth sensor (140) and the camera (150) may be implemented as a single integrated component.

[0055] According to one embodiment, the motor subsystem (160) may include a configuration for moving at least some components of the electronic device (100). For example, the motor subsystem (160) may include at least one actuator.

[0056] According to one embodiment, the motor subsystem (160) may be physically connected to at least some components of the electronic device (100) in order to move at least some components of the electronic device (100). For example, the motor subsystem (160) may be connected to at least some components of the display (130) (e.g., display panel (131)).

[0057] According to one embodiment, the motor subsystem (160) may include an inertial sensor (e.g., a 6-axis or 9-axis IMU sensor) to detect and correct errors in movement to achieve accurate movement. The inertial sensor may acquire data (e.g., acceleration, angular velocity, direction) related to the movement of a configuration connected to the motor subsystem (160).

[0058] According to one embodiment, at least one processor (110) can execute at least one instruction (121) stored in memory (120) to execute control of at least part of the electronic device (100) (e.g., adjustment of the placement of the display (130)), calculation and / or data processing.

[0059] According to one embodiment, at least one processor (110) may include at least one processing circuit and / or multiple processors. One or more of the at least one processor (110) may be configured to perform various functions described in the present disclosure individually and / or collectively. Where in the present disclosure, "processor," "at least one processor," or "one or more processors" are described as being configured to perform various functions, these terms may cover, for example, a situation in which one processor performs some of the cited functions and other processor(s) perform other parts of the cited functions, and may also cover, but are not limited to, a situation in which a single processor can perform all of the cited functions. Additionally, at least one processor (110) may include a combination of processors performing the cited / disclosed various functions, for example, in a distributed manner. At least one processor (110) may execute program instructions to achieve or perform various functions.

[0060] According to one embodiment, at least one processor (110) may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processing unit (NPU), a digital signal processor (DSP), a microcontroller unit (MCU), a sensor hub, a supplementary processor, a communication processor, an application processor, an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA), and may have a plurality of cores.

[0061] According to one embodiment, at least one processor (110) can use an artificial intelligence model to distinguish various given data, learn the relationships therewith, and optimize and / or improve the artificial intelligence model. At least one processor (110) can add training data to the artificial intelligence model to analyze data patterns and optimize the weights and / or biases of the artificial intelligence model.

[0062] According to one embodiment, at least one processor (110) can generate a depth map based on depth information obtained through a depth sensor (140) and image data (e.g., 2D pixel data) obtained through a camera (150). The depth map may be data containing depth (distance) information for each pixel.

[0063] According to one embodiment, at least one processor (110) can generate a depth map of a space containing a hand and identify a hand skeleton based on the depth map of the space containing the hand. For example, at least one processor (110) can detect the contour of the hand based on the depth map and determine the location of the joint points of the hand using a hand skeleton extraction algorithm (e.g., a deep learning-based hand skeleton extraction algorithm).

[0064] According to one embodiment, at least one processor (110) may use RGB data or other sensor data in addition to a depth map to identify the structure of the hand skeleton or determine the location of joint points.

[0065] In FIGS. 2, 3a, 3b, and 4a below, a simple and basic configuration for implementing a floating display is illustrated for ease of understanding. FIGS. 2, 3a, 3b, and 4a illustrate and describe cases in which at least one optical element (132) included in a display (e.g., the display (130) of FIG. 1) includes a retroreflector and a beam splitter. However, the display (130) according to the embodiment of the present disclosure is not limited thereto and may be implemented in a configuration that does not include a retroreflector and / or a beam splitter, or further includes various optical elements (e.g., a prism, a lens, a waveguide) for adjusting the path of light.

[0066] FIG. 2 is a drawing for explaining the configuration of a display according to one embodiment of the present disclosure.

[0067] Referring to FIG. 2, according to one embodiment, a display (e.g., display (130) of FIG. 1) may include a light source (210), a retroreflector (220), and a beam splitter (230). According to one embodiment, the light source (210) may be included in a display panel (e.g., display panel (131) of FIG. 1), and the retroreflector (220) and the beam splitter (230) may be included in an optical element (e.g., at least one optical element (132) of FIG. 1).

[0068] According to one embodiment, light emitted from a point of a light source (210) may pass through a first optical path (251) and a second optical path (261). The direction of the first optical path (251) may be the direction of the direction vector (1, tan(α)), and the direction of the second optical path (261) may be the direction of (1, tan(β)). Light passing through the first optical path (251) and the second optical path (261) may be reflected by a beam splitter (230).

[0069] According to one embodiment, according to the law of reflection, the optical paths of light reflected from the beam splitter (230) may be in the directions of (1,-tan(α)) and (1,-tan(β)), respectively. Light passing through these optical paths may be reflected from the retroreflector (220).

[0070] According to one embodiment, since the retroreflector (220) reflects the incident light in the exact opposite direction, the light paths of the light reflected from the retroreflector (220) may be in the directions of (-1,tan(α)) and (-1,tan(β)), respectively. The light reflected from the retroreflector (220) may pass through the beam splitter (230) and continue along the third light path (252) and the fourth light path (262), respectively.

[0071] According to one embodiment, the directions of the third optical path (252) and the fourth optical path (262) and the directions of the first optical path (251) and the second optical path (261) may be symmetric with respect to the beam splitter (230) (with respect to the x-axis). Accordingly, light passing through the third optical path (252) and the fourth optical path (262) may converge at a point symmetric with respect to the beam splitter (230) of the point from which it was emitted due to the symmetry. For example, the position of the image formed by the display of the structure according to FIG. 2 may be a position symmetric with respect to the beam splitter (230) of the position of the light source (210).

[0072] According to one embodiment, the size, position, and / or orientation of an image formed by a display (e.g., the display (130) of FIG. 1) may be related to the arrangement of the display (130) (e.g., a symmetrical relationship in the case of a configuration such as FIG. 2). For example, to adjust the position of the image to be formed, an electronic device (e.g., the electronic device (100) of FIG. 1) may adjust the position of at least a part of the display panel (131) and / or at least one optical element (132).

[0073] According to one embodiment, unlike FIG. 2, if the display (130) includes a more complex optical structure, the position of the image with respect to the light source or display panel may not necessarily be symmetrical. For example, if at least one optical element (132) includes a lens, the size and / or position of the image may change according to the focal length and / or magnification of the lens. For example, if at least one optical element (132) includes a prism, the size, position, and / or direction of the image may change due to the refraction of light by the prism. Even if the display (130) includes a more complex optical structure, the size, position, and / or direction of the image may be dependent on the physical arrangement of the components of the display (130) (e.g., the position or direction of the display panel (131)).

[0074] FIGS. 3a and FIGS. 3b are drawings for explaining an operation for adjusting the position of an image according to one embodiment of the present disclosure.

[0075] The configuration of the display in FIG. 3a and FIG. 3b may be the same or similar to the configuration of the display in FIG. 2.

[0076] Referring to FIG. 3a, according to one embodiment, an electronic device (e.g., the electronic device (100) of FIG. 1) can change the position of a light source from a first position (310) to a second position (311). Accordingly, since the position of the image formed is dependent on the position of the light source, the position of the image can also be changed from a third position (340) to a fourth position (341).

[0077] Referring to FIG. 3b, according to one embodiment, the electronic device (100) can change the direction of the light source from a first direction (312) to a second direction (313). Since the direction of the image formed depends on the direction of the light source, the direction of the image can also be changed from a third direction (342) to a fourth direction (343).

[0078] FIG. 4a is a drawing for explaining the relationship between the size (e.g., width) of a top and the position of the top formed according to one embodiment of the present disclosure.

[0079] The graph (420) of FIG. 4a is a graph measuring the width of an image according to the depth d from the beam splitter to the image in a display structure (410) according to one embodiment (e.g., the display structure of FIG. 2).

[0080] Referring to graph (420), according to one embodiment, a correlation close to a linear relationship can be observed between d and the width of the image. This may be because, for example, the light emitted from the light source or display panel is non-collimated light. For example, the light emitted from the light source is diverged rather than concentrated, so the width of the image increases in proportion to the length of the optical path to form the image, and the area of ​​the image increases in proportion to the square of the length of the optical path. Accordingly, d and the width of the image may have a linear relationship or a correlation close to a linear relationship.

[0081] FIG. 4b is a diagram illustrating the relationship between the size of an image (one-dimensional size such as height and width) and the projection angle of a display (e.g., the display (130) of FIG. 1) according to one embodiment of the present disclosure.

[0082] Referring to FIG. 4b, according to one embodiment, an electronic device (e.g., the electronic device (100) of FIG. 1) can adjust the arrangement of the display (130) (e.g., the arrangement of at least one optical element) so that an image is formed at a constant depth D from the display (130). In this case, the size of the image formed may be proportional to the tangent value (tan(θ)) of the projection angle θ.

[0083] According to one embodiment, a display (130) may project light in a first projection direction (431) to form a first image (441). In this case, the height of the first image (441) It could be. For example, This 45 In the case of It could be.

[0084] According to one embodiment, a display (130) may project light in a second projection direction (432) to form a second image (442). In this case, the height of the second image (442) It could be. For example, This 60 In the case of It could be.

[0085] According to one embodiment, an electronic device (e.g., the electronic device (100) of FIG. 1) can adjust the arrangement of a display (e.g., the display (130) of FIG. 1) to form an image of a specific size, thereby appropriately setting the depth and / or projection angle at which the image is formed.

[0086] The configurations illustrated in FIGS. 2, FIGS. 3, FIGS. 4a and / or FIGS. 4b are merely examples of configurations of the display of the present disclosure (e.g., the display (130) of FIG. 1), and the display (130) of the present disclosure may include any optical structure that allows an image to be formed in the air.

[0087] FIG. 5 is a drawing for explaining the operation of an electronic device (e.g., the electronic device (100) of FIG. 1) according to one embodiment of the present disclosure.

[0088] In operation 510, the electronic device (100) may acquire a first image containing a reference object. The first image may be, for example, an image acquired by photographing a space containing the reference object with a camera (e.g., the camera (150) of FIG. 1).

[0089] According to one embodiment, the electronic device (100) can identify a reference object on a first image using an object recognition algorithm. The reference object may be an object that serves as a reference for determining a location to form an image through a display (e.g., the display (130) of FIG. 1). For example, the reference object may include a hand performing a predetermined gesture.

[0090] In operation 520, the electronic device (100) can obtain first depth information, which is depth information of a space containing a reference object. The depth information may include, for example, distance information obtained through a depth sensor (e.g., the depth sensor (140) of FIG. 1).

[0091] In operation 530, the electronic device (100) can obtain a first depth map of a space containing a reference object based on a first image and first depth information. The first depth map may include, for example, pixel-wise depth information. According to one embodiment, the electronic device (100) can identify the location and / or three-dimensional shape of the reference object using the first depth map.

[0092] According to one embodiment, when the camera (150) and the depth sensor (140) are implemented in an integrated configuration (or when the camera (150) is included in the depth sensor (140)) (e.g., ToF camera), the first depth map may be described as being acquired by the depth sensor (140). According to one embodiment, when the camera (150) and the depth sensor (140) are implemented in an integrated configuration, at least some of operations 510, 520, and 530 may be omitted, at least some operations may be merged, or at least some operations may be replaced with similar operations. For example, operations 510 to 530 may be implemented as operations in which a depth map containing pixel-by-pixel depth information is acquired by the depth sensor (140) which includes camera functions.

[0093] In operation 540, the electronic device (100) can adjust the placement of the display so that an image is formed at a location relative to a reference object based on a first depth map. The location relative to the reference object may include, for example, the location of the reference object or a location adjacent to the location of the reference object. The adjustment of the display placement may include, for example, the adjustment of the position or orientation of at least a part of the display panel and / or at least one optical element included in the display.

[0094] In operation 550, the electronic device (100) may display a second image through a display. The second image may include at least one static or dynamic (e.g., video) image. The position where the image corresponding to the second image is formed may be a position relative to the reference object in operation 540.

[0095] FIG. 6 illustrates the skeleton and joint points of a hand according to one embodiment of the present disclosure.

[0096] According to one embodiment, a reference object for determining the location where an image is formed (e.g., a reference object in FIG. 5) may include a hand performing a defined gesture. For example, the reference object may include a hand with fingers extended.

[0097] According to one embodiment, an electronic device (e.g., the electronic device (100) of FIG. 1) can identify joint points and skeletons of a hand based on a depth map of a space containing a hand. For example, the electronic device (100) can identify the location of joint points and skeletal structures using a hand skeleton identification algorithm based on pixel data of an image of a hand and depth information of the hand. The hand skeleton identification algorithm may include at least one algorithm for performing operations such as contour extraction using an edge detection algorithm, skeletonization based on 3D coordinates of the hand, deep learning-based joint point estimation, and hand pose estimation.

[0098] Referring to FIG. 6, according to one embodiment, the joint points may include a first point (610), a second point (620), a third point (630), and a fourth point (640). The first point (610) corresponds to carpometacarpal joints, the second point (620) and the third point (630) correspond to metacarpophalangeal joints, and the fourth point (640) corresponds to proximal interphalangeal joints.

[0099] According to one embodiment, the skeleton of the hand can be defined based on lines connecting joint points.

[0100] FIG. 7 is a drawing for explaining the operation of determining a planar region by an electronic device (e.g., the electronic device (100) of FIG. 1) according to one embodiment of the present disclosure.

[0101] Referring to FIG. 7, the electronic device (100) can determine a display area (720) corresponding to the hand (710) based on a depth map of the hand (710) performing a defined gesture. The display area (720) may be a spatial area for forming an image by the display (130). According to one embodiment, the display area (720) may be a planar area.

[0102] According to one embodiment, the electronic device (100) can determine a display area (720) in space based on the location of the joint points of the hand (710). When there are three joint points, the plane passing through the joint points is determined as one, but when there are more than three joint points, there may not be a plane passing through all the joint points. Accordingly, the electronic device (100) can determine the display area (720) corresponding to the hand (710) using various mathematical techniques (e.g., calculation of normal vector average, principal component analysis). The determination of the display area (720) may include the determination of the location and / or orientation of the display area (720).

[0103] According to one embodiment, the electronic device (100) can determine a display area (720) based on normal vectors for planes passing through three joint points included in the joint points. For example, the electronic device (100) can obtain a normal vector using a cross product based on three joint points, and based on the average of the normal vectors obtained in the same way (hereinafter, 'average normal vector') and the average of the positions of the joint points (hereinafter, 'average point'), a plane passing through the average point and perpendicular to the average normal vector can be set to correspond to the display area (720).

[0104] According to one embodiment, for calculating the average normal vector, the planes passing through three of the n joint points may have a maximum of combinations (n, 3), but may have fewer combinations if not all joint points are used. For example, the planes passing through the first point (610), the second point (620), and the third point (630) of FIG. 6 are included when calculating the average normal vector because they are suitable for representing the plane corresponding to the palm, and the planes passing through the joint points corresponding to the proximal interphalangeal joints, such as the fourth point (640), may be excluded when calculating the average normal vector because they may not be suitable for representing the plane corresponding to the palm.

[0105] According to one embodiment, the electronic device (100) can determine the display area (720) using principal component analysis. Principal component analysis may be a statistical technique that reduces the dimensionality of data containing multiple data. In principal component analysis in three dimensions, the eigenvector corresponding to the smallest eigenvalue may be associated with a plane representing the data.

[0106] According to one embodiment, the operation of determining a display area (720) using principal component analysis based on joint points of a hand may include at least one of the following operations: an operation of obtaining a three-dimensional coordinate data set representing joint points of a hand; an operation of calculating the average coordinate of the joint points; a centering operation of obtaining a data set obtained by subtracting the average coordinate from the coordinate of each joint point; an operation of obtaining a covariance matrix from the centered data set; an operation of obtaining eigenvalues ​​and eigenvectors of the covariance matrix; and an operation of determining the eigenvector corresponding to the smallest eigenvalue as a normal vector; and an operation of defining a plane including the display area (720) based on the normal vector and the average coordinate.

[0107] According to one embodiment, in the determination operation of the display area (720) using the principal component analysis described above, the set of coordinate data may include coordinate data corresponding to all joint points, or may include coordinate data corresponding to only some joint points. For example, coordinates corresponding to the first point (610), second point (620), and third point (630) of FIG. 6 may be included in the set of coordinate data because they are suitable for representing the plane corresponding to the palm, and coordinates corresponding to proximal interphalangeal joints, such as the fourth point (640), may be excluded from the set because they may not be suitable for representing the plane corresponding to the palm.

[0108] According to one embodiment, the electronic device (100) may determine the position and / or orientation of the display area (720) using various mathematical techniques based on the position of joint points (e.g., using average normal vectors, using principal component analysis), but the size of the display area (720) (e.g., the width and height if the display area (720) is rectangular) may require a separate definition. For example, the electronic device (100) may determine the size of the display area (720) based on user input or according to an algorithm.

[0109] FIG. 8 is a drawing for explaining the operation of defining the size of a display area (e.g., the display area (720) of FIG. 7) according to one embodiment of the present disclosure.

[0110] According to one embodiment, the size of the display area (720) can be personalized according to the user. For example, the electronic device (100) can provide a convenient user experience by determining a display area of ​​a suitable size based on physical characteristics such as the user's height, arm length, hand size, and eye position.

[0111] According to one embodiment, the electronic device (100) can detect various physical characteristics of a user, such as height, eye position, and hand size, based on data obtained through a depth sensor (e.g., depth sensor (140) of FIG. 1) and / or data obtained through a camera (e.g., camera (150) of FIG. 1). For example, the electronic device (100) can identify the user's height based on a depth map of a space containing the user's entire body.

[0112] Referring to FIG. 8, according to one embodiment, the size of the display area (one-dimensional size such as height and width) may be set to be proportional to the height of the user. For example, the first size (810) of the display area may be set to be proportional to the height (815) of the first user, and the second size (820) of the display area may be set to be proportional to the height (825) of the second user.

[0113] FIG. 9 is a drawing for explaining the operation of an electronic device (e.g., the electronic device (100) of FIG. 1) according to one embodiment of the present disclosure.

[0114] In operation 910, the electronic device (100) can identify user access. For example, the electronic device (100) can identify user access using a proximity sensor. For example, the electronic device (100) can identify user access using a depth sensor (140) and / or a camera (150).

[0115] In operation 920, the electronic device (100) can identify whether the accessed user is a pre-registered user. To do this, the electronic device (100) may use user identification information stored in memory (e.g., memory (120) of FIG. 1) or an external cloud.

[0116] According to one embodiment, the electronic device (100) can identify whether the user is a pre-registered user based on data about the user's face. The data about the face may include, for example, a depth map and / or 2D pixel data.

[0117] In operation 930, the electronic device (100) can identify whether the user performs a hand gesture to define a display area based on identifying that the user is a pre-registered user. Whether a gesture is performed can be identified, for example, using a depth map.

[0118] In operation 941, when a user performs the gesture, the electronic device (100) may determine a display area based on the skeleton of the hand performing the gesture. The skeleton of the hand may be obtained, for example, based on the first depth map in FIG. 5. The determination of the display area may be performed, for example, through the mathematical technique described in FIG. 7.

[0119] In operation 951, the electronic device (100) can adjust the placement of the display (e.g., the display (130) of FIG. 1) so that an image is formed in a determined display area. The position, orientation, and / or size of the image formed by the display (130) may depend on the placement of the display (130) (e.g., the position or orientation of the display panel (131)).

[0120] In operation 942, if the user does not perform the gesture, the electronic device (100) can adjust the arrangement of the display (130) based on pre-stored setting information mapped to the user's identification information. The setting information may include, for example, setting information regarding the arrangement of the display (130) that has been set and stored by the user performing the gesture in the past.

[0121] In operation 960, the electronic device (100) can provide guidance for performing a gesture for defining a display area based on identifying that the user is a user that has not been previously registered, and can store the user's identification information.

[0122] In operation 970, assuming that the user performs the gesture after providing guidance on performing the gesture, the electronic device (100) may determine a display area based on the skeleton of the user's hand performing the gesture. The skeleton of the hand may be obtained, for example, based on the first depth map in FIG. 5. The determination of the display area may be performed, for example, through the mathematical technique described in FIG. 7.

[0123] In operation 980, the electronic device (100) can adjust the placement of the display (130) so that an image is formed in the determined display area. At this time, the electronic device (100) can map the settings related to the placement of the display (130) to the user's identification information and store them in memory (e.g., memory (120) of FIG. 1) or an external cloud.

[0124] In operation 990, the electronic device (100) can display a second image (e.g., the second image in FIG. 5) through the display (130).

[0125] FIG. 10 illustrates a situation in which an electronic device (e.g., the electronic device (100) of FIG. 1) according to one embodiment of the present disclosure recognizes a touch input.

[0126] According to one embodiment, an electronic device (100) may obtain a second depth map of an image (1020) formed by a display (e.g., the display (130) of FIG. 1) and a second depth map of a touch input object (1010). For example, based on the second depth map and the third depth map, the electronic device (100) may obtain the location of the image (1020) (e.g., depth value per pixel) and the location of the touch input object (1010). The touch input object (1010) is illustrated as a hand, but the form of the touch input object of the present disclosure is not limited thereto.

[0127] According to one embodiment, an electronic device (100) can recognize a touch input when the positional relationship between a touch input object (1010) and a surface (1020) satisfies at least one condition. The at least one condition may include, for example, a condition in which the touch input object (1010) contacts the surface (1020), a condition in which the distance of the touch input object (1010) from the surface (1020) is less than or equal to a threshold value, a condition in which the depth through which the touch input object (1010) penetrates the surface (1020) is greater than or equal to a threshold value, and / or a condition in which at least one of the above conditions is continuously satisfied for a threshold time or longer.

[0128] Referring to FIG. 10, according to one embodiment, when the depth to which a touch input object (1010) penetrates the surface (1020) is equal to the touch depth (1030), the electronic device (100) may recognize the touch input if the touch depth (1030) is greater than or equal to a predetermined threshold, and may not recognize the touch input if it is less than or equal to a predetermined threshold.

[0129] FIG. 11 illustrates a motor subsystem according to one embodiment of the present disclosure.

[0130] According to one embodiment, the motor subsystem (1100) may be a component included in an electronic device (e.g., the electronic device (100) of FIG. 1). For example, the motor subsystem (1100) may be the motor subsystem (160) of FIG. 1.

[0131] According to one embodiment, the motor subsystem (1100) may be a configuration connected to at least some of the components of the display (130) to move at least some of the components of the display (130). The motor subsystem (1100) may include, for example, at least one actuator (1100) and / or an inertial measurement unit (IMU) sensor (1120).

[0132] According to one embodiment, the actuator may be a device that generates physical motion (e.g., movement, rotation, vibration, etc.) using electrical, mechanical, or electromagnetic energy. At least one actuator (1110) may be connected to at least a part of the display panel (131) and / or at least one optical element (132) and used to adjust the position of an image formed in the air or the position of a focus.

[0133] According to one embodiment, the IMU sensor (1120) may be a sensor that measures acceleration and angular velocity to track changes in the position, velocity, and / or direction of an object in 3D space. The electronic device (100) may move a configuration attached by at least one actuator (1110) while simultaneously acquiring motion information with the IMU sensor (1120). The motion information may include, for example, position errors that occur while the motor subsystem (1100) is operating. The motion information may be used, for example, to detect excessive movement (e.g., acceleration above a threshold value).

[0134] According to one embodiment, the electronic device (100) can perform correction for precise movement using position error detected through the IMU sensor (1120). According to one embodiment, if the electronic device (100) detects excessive movement, it can mitigate it (e.g., reduce speed or acceleration) to prevent damage to at least some components of the connected display (130).

[0135] In addition to the configuration shown in FIG. 11, the motor subsystem according to the embodiment of the present disclosure may include all configurations for moving at least some of the components of the display (130).

[0136] FIGS. 12a and FIGS. 12b illustrate examples of implementations of at least one actuator (e.g., at least one actuator (1100) of FIG. 11) according to embodiments of the present disclosure.

[0137] According to one embodiment, at least one actuator (1100) may include three or more actuators or an actuator combined with at least one ball joint. One actuator combined with three or more actuators or ball joints may move a connected component linearly (e.g., up and down) or change its direction (e.g., roll, pitch, yaw).

[0138] According to one embodiment, at least one actuator (1100) may be connected to a first component (1250) included in a display (e.g., the display (130) of FIG. 1). The first component (1250) may be, for example, a display panel or an optical element. In FIG. 12a and FIG. 12b, the first component (1250) is depicted in a rectangular shape, but the present disclosure is not limited thereto.

[0139] Referring to FIG. 12a, according to one embodiment, at least one actuator (1100) may include a first actuator (1211), a second actuator (1212), a third actuator (1213), and a fourth actuator (1214).

[0140] According to one embodiment, a first actuator (1211), a second actuator (1212), a third actuator (1213), and a fourth actuator (1214) may be connected to a first configuration (1250) and used to control the movement of the first configuration (1250). For example, if the first actuator (1211), the second actuator (1212), the third actuator (1213), and the fourth actuator (1214) all move up or all move down, the first configuration (1250) may move linearly up or down. For example, if the first actuator (1211) and the second actuator (1212) move upward and the third actuator (1213) and the fourth actuator (1214) move downward, the first configuration (1250) can be tilted.

[0141] Referring to FIG. 12b, according to one embodiment, at least one actuator (1100) may include a fifth actuator (1220). The fifth actuator (1220) may be coupled with a ball joint (1230). The ball joint (1230) may be, for example, a connecting structure designed to be rotatable in multiple directions.

[0142] According to one embodiment, the fifth actuator (1220) may be connected to the first component (1250) and used to control the movement of the first component (1250). For example, if the fifth actuator (1220) moves up and down or rotates due to the ball joint (1230), the first component (1250) may also move up and down or rotate in response.

[0143] According to one embodiment of the present disclosure, an electronic device may be provided. The electronic device may include: a display comprising a display panel and at least one optical element; a depth sensor; a memory comprising at least one storage medium for storing instructions; and at least one processor comprising a processing circuit. The at least one processor may perform at least one operation by executing instructions.

[0144] According to one embodiment, the at least one processor may be configured to: acquire a first depth map of a space containing a reference object through the depth sensor, wherein the first depth map includes pixel-wise depth information; adjust the arrangement of the display so that an image is formed at a position in space related to the reference object based on the first depth map, wherein the arrangement of the display includes changing the position or orientation of at least one of the display panel or the at least one optical element; and display an image through the display.

[0145] According to one embodiment, the reference object includes a hand that performs a defined gesture, and the at least one processor may be configured to: identify the locations of a plurality of joint points of the hand that performs the gesture based on the first depth map; determine a display area in space based on the locations of the plurality of joint points; and adjust the arrangement of the display so that an image is formed at the location of the display area.

[0146] According to one embodiment, the plurality of joint points includes three or more joint points, and the at least one processor is configured to: obtain a normal vector of at least one plane based on the positions of the plurality of joint points, and the at least one plane includes a plane through which three joint points included in the plurality of joint points pass; and determine the position and orientation of the display area based on the at least one normal vector for the at least one plane and the positions of the plurality of joint points, and the display area may be a planar area.

[0147] According to one embodiment, the at least one processor is configured to: obtain a covariance matrix based on the positions of the plurality of joint points; obtain eigenvalues ​​of the covariance matrix and eigenvectors corresponding to the eigenvalues; and determine the position and orientation of the display area based on the eigenvector corresponding to the smallest eigenvalue among the eigenvalues ​​and the positions of the plurality of joint points, wherein the display area may be a planar area.

[0148] According to one embodiment, the at least one processor is configured to adjust the arrangement of the display to adjust the size of the image to be formed as the image is displayed, and the size of the image to be formed may be related to at least one of the depth at which the image is formed or the projection angle of the display.

[0149] According to one embodiment, the at least one processor may be configured to: identify whether the user is a pre-registered user based on identifying the user; provide guidance regarding the performance of the gesture based on identifying that the user is a pre-registered user and store the user's identification information; determine a display area in space based on the positions of the plurality of joint points of the hand performing the gesture; and adjust the arrangement of the display so that an image is formed in the display area.

[0150] According to one embodiment, the at least one processor may be configured to: identify whether the user performs the gesture based on identifying that the user is a pre-registered user; determine a display area in space based on the positions of the plurality of joint points of the hand performing the gesture based on whether the user performs the gesture, and adjust the arrangement of the display so that an image is formed in the display area; and adjust the arrangement of the display based on setting information stored in association with the user's identification information based on whether the user does not perform the gesture.

[0151] According to one embodiment, the at least one processor is configured to identify whether the user is a pre-registered user based on data regarding the user's face, and the data regarding the face may include at least one of a depth map or 2D pixel data.

[0152] According to one embodiment, the at least one processor may be configured to: identify user access; identify body information of the user, wherein the body information includes the height of the user; and, based on the body information of the user, adjust the size of the image to be formed as the image is displayed.

[0153] According to one embodiment, the at least one processor comprises: acquiring a second depth map of an image formed by displaying the image and a third depth map of a touch input object, wherein the second depth map and the third depth map include pixel-wise depth information; identifying whether the positional relationship between the image and the touch input object satisfies at least one condition based on the second depth map and the third depth map; and being configured to recognize a touch input based on identifying that the at least one condition is satisfied, wherein the at least one condition may include at least one of a condition in which the touch input object contacts the image, a condition in which the distance of the touch input object from the image is less than or equal to a first threshold, a condition in which the depth through which the touch input object penetrates the image is greater than or equal to a second threshold, or a condition in which at least one of the conditions is continuously satisfied for a threshold time or longer.

[0154] According to one embodiment of the present disclosure, a method for operating an electronic device may be provided. The method comprises: acquiring a first depth map of a space including a reference object, wherein the first depth map includes pixel-wise depth information; adjusting the arrangement of a display of the electronic device so that an image is formed at a position in space related to the reference object based on the first depth map; and displaying a second image through the display; wherein the display may include a display panel and at least one optical element.

[0155] According to one embodiment, the reference object includes a hand performing a defined gesture, and the method may further include: an action of identifying the positions of a plurality of joint points of the hand performing the gesture based on the first depth map; an action of determining a display area in space based on the positions of the plurality of joint points; and an action of adjusting the arrangement of the display so that an image is formed at the position of the display area.

[0156] According to one embodiment, the plurality of joint points includes three or more joint points, and the operation of determining a display area in space based on the positions of the plurality of joint points may include: obtaining a normal vector of at least one plane based on the positions of the plurality of joint points, wherein the at least one plane includes a plane through which three joint points included in the plurality of joint points pass; and determining the position and direction of the display area based on at least one normal vector for the at least one plane and the positions of the plurality of joint points, wherein the display area is a planar area.

[0157] According to one embodiment, the operation of determining a display area in space based on the positions of the plurality of joint points may include: the operation of obtaining a covariance matrix based on the positions of the plurality of joint points; the operation of obtaining eigenvalues ​​of the covariance matrix and eigenvectors corresponding to the eigenvalues; and the operation of determining the position and direction of the display area based on the eigenvector corresponding to the smallest eigenvalue among the eigenvalues ​​and the positions of the plurality of joint points, wherein the display area is a planar area.

[0158] According to one embodiment, the method further includes an operation of adjusting the arrangement of the display to adjust the size of the image to be formed as the second image is displayed, and the size of the image to be formed may be related to at least one of the depth at which the image is formed or the projection angle of the display.

[0159] According to one embodiment, the method may further include: an action of identifying whether the user is a pre-registered user based on identifying the user; an action of providing guidance on performing the gesture and storing the user's identification information based on identifying that the user is not a pre-registered user; an action of determining a display area in space based on the positions of the plurality of joint points of the hand performing the gesture; and an action of adjusting the arrangement of the display so that an image is formed in the display area.

[0160] According to one embodiment, the method may further include: an action of identifying whether the user performs the gesture based on identifying that the user is a pre-registered user; an action of determining a display area in space based on the positions of the plurality of joint points of the hand performing the gesture and adjusting the arrangement of the display so that an image is formed in the display area when the user performs the gesture; and an action of adjusting the arrangement of the display based on pre-stored setting information mapped to the user's identification information when the user does not perform the gesture.

[0161] According to one embodiment, the method may further include an operation to identify whether the user is a pre-registered user based on data regarding the user's face, wherein the data regarding the face includes at least one of a depth map or 2D pixel data.

[0162] According to one embodiment, the method further comprises: an operation of identifying user access; and an operation of identifying the user’s body information, wherein the body information includes the user’s height; and the operation of adjusting the arrangement of the display to adjust the size of the image to be formed by displaying the second image may include an operation of adjusting the size of the image to be formed by displaying the second image based on the user’s body information.

[0163] According to one embodiment, the method further comprises: an operation of obtaining a second depth map of an image formed by displaying the second image and a third depth map of a touch input object, wherein the second depth map and the third depth map include pixel-wise depth information; an operation of identifying whether the positional relationship between the image and the touch input object satisfies at least one condition based on the second depth map and the third depth map; and an operation of recognizing a touch input based on identifying that the at least one condition is satisfied, wherein the at least one condition may include at least one of a condition in which the touch input object contacts the image, a condition in which the distance of the touch input object from the image is less than or equal to a first threshold, a condition in which the depth through which the touch input object penetrates the image is greater than or equal to a second threshold, or a condition in which at least one of the above conditions is continuously satisfied for a threshold time or longer.

[0164] Meanwhile, the various embodiments described above may be implemented as software containing instructions stored on a device-readable storage medium, included in a computer program product in the form of a device-readable storage medium (e.g., flash memory, SSD, HDD, optical disc, magnetic tape, etc.), or distributed online through an application store, website, or cloud service. Additionally, they may be implemented within a recording medium readable by a computer or similar device using software, hardware, firmware, or a combination thereof.

[0165] Each component according to the various embodiments described above may be composed of a single or multiple entities, and some auxiliary components may be omitted or additionally included. Some components may be integrated into a single entity to perform the same or similar functions as those performed by each corresponding component prior to integration.

[0166] Each component according to the various embodiments described above may be composed of a single or multiple entities, and some auxiliary components may be omitted or additionally included. Some components may be integrated into a single entity to perform the same or similar functions as those performed by each corresponding component prior to integration.

[0167] The operations according to the various embodiments described above may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

Claims

1. In an electronic device, A display comprising a display panel and at least one optical element; Depth sensor; Memory comprising at least one storage medium for storing instructions; and at least one processor including a processing circuit; comprising, The above at least one processor is: Through the depth sensor, a first depth map of a space containing a reference object is obtained, and the first depth map includes pixel-wise depth information; Based on the first depth map above, the arrangement of the display is adjusted so that an image is formed at a spatial position related to the reference object, and the arrangement of the display includes changing the position or orientation of at least one of the display panel or the at least one optical element; A display configured to display an image through the above display, Electronic device.

2. In Paragraph 1, The above reference object includes a hand that performs a defined gesture, and The above at least one processor is: Based on the first depth map above, the locations of a plurality of joint points of the hand performing the gesture are identified; Determining a display area in space based on the positions of the above-mentioned plurality of joint points; Configured to adjust the arrangement of the display so that an image is formed at the location of the display area, Electronic device.

3. In Paragraph 2, The above plurality of joint points include three or more joint points, and The above at least one processor is: A normal vector of at least one plane is obtained based on the positions of the plurality of joint points, and the at least one plane includes a plane through which three joint points included in the plurality of joint points pass; The position and orientation of the display area are determined based on at least one normal vector for the at least one plane and the positions of the plurality of joint points, wherein the display area is a planar area. Electronic device.

4. In Paragraph 2, The above at least one processor is: A covariance matrix is ​​obtained based on the positions of the above plurality of joint points; Obtain the eigenvalues ​​of the above covariance matrix and the eigenvectors corresponding to the eigenvalues; It is configured to determine the position and orientation of the display area based on the eigenvector corresponding to the smallest eigenvalue among the above eigenvalues ​​and the positions of the plurality of joint points, wherein the display area is a planar area. Electronic device.

5. In any one of paragraphs 1 through 4, The above-mentioned at least one processor is, It is configured to adjust the arrangement of the display to adjust the size of the image to be formed as the above image is displayed, and The size of the image to be formed is related to at least one of the depth at which the image is formed or the projection angle of the display, Electronic device.

6. In any one of paragraphs 2 through 4, The above at least one processor is: Based on identifying the user, determine whether the user is a pre-registered user; Based on identifying that the above user is a user who has not been previously registered, providing guidance on the performance of the above gesture and storing the identification information of the above user; Determining a display area in space based on the positions of the plurality of joint points of the hand performing the above gesture; A configuration configured to adjust the arrangement of the display so that an image is formed in the display area. Electronic device.

7. In Paragraph 6, The above at least one processor is: Based on identifying that the above user is a pre-registered user, identifying whether the above user performs the above gesture; Based on the user performing the gesture, a display area in space is determined based on the positions of the plurality of joint points of the hand performing the gesture, and the arrangement of the display is adjusted so that an image is formed in the display area; Based on the fact that the user does not perform the gesture, the arrangement of the display is configured to be adjusted based on the setting information stored in association with the user's identification information. Electronic device.

8. In Paragraph 6 or 7, The above-mentioned at least one processor is, Based on data regarding the face of the user, the system is configured to identify whether the user is a pre-registered user, wherein the data regarding the face includes at least one of a depth map or 2D pixel data. Electronic device.

9. In Paragraph 5, The above at least one processor is: Identify user access; Identifying the body information of the above user, and the body information includes the height of the above user; Configured to adjust the size of the image to be formed by displaying the image based on the body information of the user. Electronic device.

10. In any one of paragraphs 1 through 9, The above at least one processor is: A second depth map of the image formed by displaying the above image and a third depth map of the touch input object are obtained, and the second depth map and the third depth map include pixel-wise depth information; Based on the second depth map and the third depth map, identify whether the positional relationship between the image and the touch input object satisfies at least one condition; It is configured to recognize touch input based on identifying that at least one of the above conditions is satisfied, and The above at least one condition includes at least one of the following: a condition in which the touch input object contacts the surface; a condition in which the distance of the touch input object from the surface is less than or equal to a first threshold value; a condition in which the depth of penetration of the touch input object into the surface is greater than or equal to a second threshold value; or a condition in which at least one of the above conditions is continuously satisfied for a threshold time or longer. Electronic device.

11. In a method of operating an electronic device: An operation to obtain a first depth map of a space containing a reference object, wherein the first depth map includes pixel-wise depth information; An operation to adjust the arrangement of the display of the electronic device so that an image is formed at a spatial position related to the reference object based on the first depth map; and The operation of displaying a second image through the above display; is included, The above display comprises a display panel and at least one optical element, method.

12. In Paragraph 11, The above reference object includes a hand that performs a defined gesture, and The above method is: An action of identifying the positions of a plurality of joint points of a hand performing the gesture based on the first depth map; An operation of determining a display area in space based on the positions of the plurality of joint points; and The operation of adjusting the arrangement of the display so that an image is formed at the location of the display area; further comprising method.

13. In Paragraph 11 or 12, The method further includes the operation of adjusting the arrangement of the display to adjust the size of the image to be formed by displaying the second image; The size of the image to be formed is related to at least one of the depth at which the image is formed or the projection angle of the display, method.

14. In any one of paragraphs 12 to 13, An operation to identify whether the user is a pre-registered user based on identifying the user; An action of providing guidance on the performance of the gesture and storing the user's identification information based on identifying that the user is a user who has not been previously registered; An action of determining a display area in space based on the positions of the plurality of joint points of the hand performing the gesture; and The operation of adjusting the arrangement of the display so that an image is formed in the display area; further comprising method.

15. In any one of paragraphs 11 through 14, An operation to obtain a second depth map of an image formed by displaying the second image and a third depth map of a touch input object, wherein the second depth map and the third depth map include pixel-wise depth information; An operation to identify whether the positional relationship between the image and the touch input object satisfies at least one condition based on the second depth map and the third depth map; and The operation of recognizing touch input based on identifying that at least one of the above conditions is satisfied; further comprising, The above at least one condition includes at least one of the following: a condition in which the touch input object contacts the surface; a condition in which the distance of the touch input object from the surface is less than or equal to a first threshold value; a condition in which the depth of penetration of the touch input object into the surface is greater than or equal to a second threshold value; or a condition in which at least one of the above conditions is continuously satisfied for a threshold time or longer. method.

Images