Electronic device and control method therefor

Abstract

This electronic device comprises: a camera; a drive unit configured to move the electronic device; a projection unit for outputting light from a light source to a projection surface positioned outside the electronic device; a memory configured to store instructions; and one or more processors configured to execute the stored instructions collectively or individually. The stored instructions are configured to: project a UI including a plurality of UI elements onto the projection surface through the projection unit; acquire, through the camera, an image related to a motion of a user indicating a selection of the user made for one of the plurality of UI elements; obtain, from the acquired image, information corresponding to the distances between the electronic device and feature areas related to a foot of the user; and, on the basis of the obtained information, select a projected UI element indicated to be selected by the user by the motion of the user from among the plurality of UI elements.

Description

Electronic device and control method thereof

[0001] The present disclosure relates to an electronic device and a method for controlling the same, and more specifically, to an electronic device and a method for controlling the same for detecting the movement of a user.

[0002] An artificial intelligence system is a computer system that implements human-level intelligence, in which machines learn and make judgments on their own, and whose recognition rate improves with use.

[0003] Artificial intelligence technology consists of machine learning (deep learning) technology, which utilizes algorithms to self-classify and learn the characteristics of input data, and component technologies that mimic the functions of the human brain, such as cognition and judgment, by utilizing machine learning algorithms.

[0004] The elemental technologies may include, for example, at least one of linguistic understanding technology that recognizes human language / characters, visual understanding technology that perceives objects like human vision, inference / prediction technology that judges information to logically reason and predict, knowledge representation technology that processes human experience information into knowledge data, and motion control technology that controls autonomous driving of vehicles and the movement of robots. In particular, visual understanding is a technology that perceives and processes objects like human vision, and includes object recognition, object tracking, image search, person recognition, scene understanding, spatial understanding, image enhancement, etc. Additionally, inference and prediction is a technology that judges information to logically reason and predict, and includes knowledge / probability-based inference, optimization prediction, preference-based planning, recommendation, etc.

[0005] Meanwhile, in recent years, electronic devices that move within specific spaces (e.g., homes, restaurants, etc.) and perform various functions are being developed. For instance, various electronic devices such as mobile projectors, robotic vacuum cleaners, serving robots, and guide robots are performing diverse functions while moving within a specific space.

[0006] In the case of portable electronic devices such as portable projectors, users with limited mobility, such as people with disabilities, faced difficulties in inputting user commands to control the device due to restrictions on movement.

[0007] According to one embodiment of the present disclosure, an electronic device comprises: a camera; a driving unit configured to move the electronic device; a projection unit that outputs light from a light source to a projection surface located outside the electronic device; a memory configured to store instructions; and one or more processors configured to execute the stored instructions collectively or individually. The stored instructions may be configured to project a UI including a plurality of UI elements onto the projection surface through the projection unit, acquire an image related to user movement indicating a user's selection of one of the plurality of UI elements through the camera, acquire information corresponding to the distance between the electronic device and a feature region related to the user's foot from the acquired image, and select a projected UI element among the plurality of UI elements that appears to have been selected by the user due to the user's movement based on the acquired information.

[0008] The above stored instructions may be configured to determine whether the user is seated on a wheelchair, and if the user is identified as being seated on a wheelchair, to identify the user as a wheelchair user, and to acquire a feature region related to the user's feet while the user is identified as a wheelchair user.

[0009] The above feature region may include: a first feature region corresponding to a first frame of the wheelchair corresponding to the right foot of the wheelchair user and a support of the wheelchair for the right foot of the wheelchair user; and a second feature region corresponding to a second frame of the wheelchair corresponding to the left foot of the wheelchair user and a support of the wheelchair for the left foot of the wheelchair user.

[0010] The above stored instructions may be configured to acquire information regarding the direction in which the wheelchair user has rotated as the rotation direction of the wheelchair user based on the acquired information corresponding to the distance between the feature area and the electronic device, and to select a UI element among the plurality of UI elements that corresponds to the user's rotation direction based on the acquired information regarding the rotation direction.

[0011] The above stored instructions may be configured to identify a counterclockwise rotation of the user if the first distance between the first feature area and the electronic device is shorter than the second distance between the second feature area and the electronic device, and to identify a clockwise rotation of the user if the first distance is longer than the second distance.

[0012] The above stored instructions may be configured to obtain a correction value based on the difference between the first distance between the first feature area and the electronic device and the second distance between the second feature area and the electronic device before the wheelchair user moves, and to select the projected UI element among the plurality of UI elements based on the obtained correction value and the obtained information.

[0013] The above stored instructions may be configured to acquire an image of the user through the camera before the user moves, and, based on the acquired image before the user moves, to perform keystone correction to make the UI parallel to the user if the projected UI is not parallel to the user.

[0014] The above stored instructions may be configured to acquire an image of the user through the camera before the user moves, and if the UI and the user are not parallel based on the acquired image before the user moves, to drive the electronic device through the driving unit so that the user and the UI become parallel.

[0015] The above stored instructions may be configured such that when a first event corresponding to the projected UI is detected, which is related to a first user among multiple users who is not in a wheelchair, the electronic device is driven to an area corresponding to the first user through the driving unit, the UI is projected to an area corresponding to the first user through the projection unit, and information about the first user's dominant foot is obtained based on the first user's foot movement.

[0016] The above stored instructions may be configured such that when a second event related to the first user and corresponding to the projected UI is detected while information corresponding to the first user's primary foot is obtained, the electronic device is driven to an area corresponding to the first user through the driving unit, and a modified UI in which the arrangement of the plurality of UI elements is changed based on information regarding the first user's primary foot is projected through the projection unit.

[0017] A method for controlling an electronic device according to one embodiment of the present disclosure, comprising a camera, a driving unit configured to move the electronic device, a projection unit that outputs light from a light source to a projection surface located outside the electronic device, a memory configured to store instructions, and one or more processors configured to execute the stored instructions collectively or individually, wherein the control method may include: a step of projecting a UI including a plurality of UI elements onto the projection surface through the projection unit by the at least one processor; a step of acquiring an image related to a user's movement indicating a user's selection of one of the plurality of UI elements through the camera; a step of acquiring information corresponding to the distance between the electronic device and a feature region related to the user's foot from the acquired image; and a step of selecting a projected UI element among the plurality of UI elements that appears to have been selected by the user due to the user's movement based on the acquired information.

[0018] The above control method may include: a step of determining whether the user is seated on a wheelchair; a step of identifying the user as a wheelchair user when the user is identified as being seated on a wheelchair; and a step of acquiring a feature region related to the user's foot while the user is identified as the wheelchair user.

[0019] The above feature region may include: a first feature region corresponding to a first frame of the wheelchair corresponding to the right foot of the wheelchair user and a support of the wheelchair for the right foot of the wheelchair user; and a second feature region corresponding to a second frame of the wheelchair corresponding to the left foot of the wheelchair user and a support of the wheelchair for the left foot of the wheelchair user.

[0020] The above-mentioned selection step may include: a step of obtaining information regarding the direction in which the wheelchair user has rotated as the direction of rotation of the wheelchair user based on the obtained information corresponding to the distance between the feature area and the electronic device; and a step of selecting a UI element corresponding to the direction of rotation of the user among the plurality of UI elements based on the information regarding the obtained direction of rotation.

[0021] The above identifying step identifies the user's rotation in a counterclockwise direction if the first distance between the first feature area and the electronic device is shorter than the second distance between the second feature area and the electronic device, and identifies the user's rotation in a clockwise direction if the first distance is longer than the second distance.

[0022] The above control method includes the step of obtaining a correction value based on the difference between the first distance between the first feature area and the electronic device and the second distance between the second feature area and the electronic device before the user moves; and the selecting step may select one of the plurality of UI elements based on information corresponding to the correction value and the distance.

[0023] The above control method may include the step of performing keystone correction to make the UI correspond to a parallel state with the user based on an image captured through the camera before the user moves, if the UI and the user do not correspond to a parallel state.

[0024] The above control method may include the step of driving the electronic device so that the user and the UI correspond to a parallel state based on an image captured through the camera before the user moves, if the UI and the user do not correspond to a parallel state.

[0025] The above control method may include: a step of driving to an area corresponding to the first user based on the first event when a first event corresponding to UI projection is detected based on a first user who is not in a wheelchair among a plurality of users; a step of obtaining information corresponding to the first user's dominant foot based on the foot movement of the first user while projecting the UI to the area corresponding to the first user; a step of driving to an area corresponding to the first user based on the second event when a second event for projecting the UI based on the first user is detected after the information regarding the first user's dominant foot is obtained; and a step of projecting a modified UI in which the arrangement of UI elements included in the UI is changed based on the information regarding the first user's dominant foot.

[0026] In a non-transient computer-readable medium storing instructions for executing a method of controlling an electronic device including a camera and a projection unit according to one embodiment of the present disclosure, the method of controlling the electronic device comprises: a step of projecting a UI including a plurality of UI elements through the projection unit; a step of acquiring an image related to a user’s movement corresponding to the selection of one of the plurality of UI elements through the camera; a step of acquiring information corresponding to the distance between a plurality of feature regions related to the user’s foot and the electronic device from the image; and a step of selecting one of the plurality of UI elements based on the information corresponding to the distance.

[0027] FIG. 1 is a drawing illustrating an electronic device for receiving user commands from a user in a wheelchair according to one embodiment of the present disclosure.

[0028] FIG. 2 is a block diagram showing the configuration of an electronic device according to one embodiment of the present disclosure,

[0029] FIGS. 3a and 3b are drawings illustrating the external appearance of an electronic device according to various embodiments of the present disclosure.

[0030] FIG. 4 is a flowchart illustrating an embodiment for selecting a UI element based on a user's foot movement according to one embodiment of the present disclosure,

[0031] FIG. 5 is a drawing for explaining a plurality of feature regions according to one embodiment of the present disclosure,

[0032] FIGS. 6a and 6b are drawings for illustrating an embodiment of selecting a UI element based on a user's foot movement according to an embodiment of the present disclosure.

[0033] FIGS. 7a to 8c are drawings illustrating a method for correcting a UI so that the user and the UI are parallel, according to an embodiment of the present disclosure.

[0034] FIG. 9 is a flowchart illustrating a control method for an electronic device capable of providing a UI corresponding to the characteristics of a plurality of users, according to one embodiment of the present disclosure.

[0035] FIGS. 10a to 11b are drawings for explaining a method of correcting a UI according to user characteristics according to one embodiment of the present disclosure.

[0036] The embodiments described herein are subject to various modifications and may have various forms; specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope of specific embodiments and should be understood to include various modifications, equivalents, and / or alternatives of the embodiments of the present disclosure. In relation to the description of the drawings, similar reference numerals may be used for similar components.

[0037] In describing the present disclosure, if it is determined that a detailed description of related known functions or configurations could unnecessarily obscure the essence of the present disclosure, such detailed description is omitted.

[0038] Additionally, the following embodiments may be modified in various other forms, and the scope of the technical concept of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to make the present disclosure more faithful and complete and to fully convey the technical concept of the present disclosure to those skilled in the art.

[0039] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit the scope of the rights. The singular expression includes the plural expression unless the context clearly indicates otherwise.

[0040] In the present disclosure, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of such features (e.g., numerical values, functions, actions, or components such as parts) and do not exclude the presence of additional features.

[0041] In the present disclosure, expressions such as “A or B,” “at least one of A or / and B,” or “one or more of A or / and B” may include all possible combinations of items listed together. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” may refer to cases including (1) at least one A, (2) at least one B, or (3) both at least one A and at least one B.

[0042] Expressions such as "first," "second," "first," or "second" used in this disclosure may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit said components.

[0043] Where it is stated that a certain component (e.g., a first component) is "(operatively or communicatively) coupled with / to" or "connected to" another component (e.g., a second component), it should be understood that the said certain component may be directly connected to the said other component or connected through another component (e.g., a third component).

[0044] On the other hand, when it is stated that a certain component (e.g., a first component) is "directly connected" or "directly coupled" to another component (e.g., a second component), it may be understood that no other component (e.g., a third component) exists between said certain component and said other component.

[0045] As used in this disclosure, the expression “configured to” may be replaced, depending on the context, with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” may not necessarily mean only “specifically designed to” in hardware.

[0046] Instead, in some situations, the expression “device configured to do something” may mean that the device is “capable of doing something” together with other devices or components. For example, the phrase “processor configured (or set) to perform A, B, and C” may mean a dedicated processor for performing those operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or application processor) capable of performing those operations by executing one or more software programs stored in a memory device.

[0047] In the embodiments, a 'module' or 'part' performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Additionally, a plurality of 'modules' or a plurality of 'parts' may be integrated into at least one module and implemented by at least one processor, except for the 'module' or 'part' that needs to be implemented in specific hardware.

[0048] In the embodiments, "state" may be replaced with terms such as "situation," "mode," etc. For example, "wall projection state" may be replaced with "wall projection situation," "wall projection mode," etc.

[0049] Meanwhile, various elements and areas in the drawings are depicted schematically. Accordingly, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0050]

[0051] FIG. 1 is a drawing illustrating an electronic device for receiving user commands from a user in a wheelchair, according to one embodiment of the present disclosure. According to one embodiment of the present disclosure, the electronic device (100) may be implemented as a mobile projector as shown in FIG. 1, but this is merely one embodiment and may be implemented as various mobile electronic devices such as a robot.

[0052] In particular, according to one embodiment of the present disclosure, the electronic device (100) may include a projector for projecting an image. In this case, the projector is a device that projects an image onto a screen, wall, or floor through light. The projector can process an input image signal and project the processed image signal by magnifying it through a light source and a lens system.

[0053] In particular, the projector may include a long-focus projection lens and an ultra-short-focus projection lens. The projector can project an image to a location far from a wall or a screen installed on the wall using the long-focus projection lens. Additionally, the projector can project an image to a location close to a wall or the floor using the ultra-short-focus projection lens.

[0054] According to one embodiment of the present disclosure, an electronic device (100) may move toward a user and project (or provide, output, etc.) a UI around the user (e.g., in front of the user). Here, the UI may include a plurality of UI elements for the electronic device (100) to receive user commands from the user.

[0055] In one or more embodiments, the electronic device (100) can select one of a plurality of UI elements according to the movement of the user's hand or foot. For example, if a movement of the user's hand or foot moving to the left is detected, the electronic device (100) can select a UI element placed on the left. Additionally, if a movement of the user's hand or foot moving counterclockwise is detected, the electronic device (100) can select a UI element placed on the right.

[0056] In one or more embodiments, the electronic device (100) can identify a user with limited mobility (10) (e.g., a user in a wheelchair) as shown in FIG. 1. When an event is detected to project a UI (20) onto the user in the wheelchair (10), the electronic device (100) can project the UI (20) in front of the user (10) as shown in FIG. 1. Here, the UI (20) may include a first UI element (e.g., later) and a second UI element (e.g., Okay).

[0057] In one or more embodiments, the electronic device (100) may acquire an image (e.g., a depth image) to obtain information about a user's movement of selecting one of a plurality of UI elements through a camera (e.g., a depth camera). The electronic device (100) may extract a plurality of feature regions related to the user's feet from the image. In one or more embodiments, the plurality of feature regions may include a first feature region corresponding to a first frame of a wheelchair and a right footrest corresponding to the user's right foot, and a second feature region corresponding to a second frame of a wheelchair and a left footrest corresponding to the user's left foot.

[0058] The electronic device (100) can obtain information corresponding to the distance between a plurality of feature regions and the electronic device (100). Additionally, the electronic device (100) can identify information regarding a user's movement (e.g., rotation direction, etc.) based on the information corresponding to the distance, and can select one of a plurality of UI elements according to the identified user's movement.

[0059] As described above, by detecting the movements of users with limited mobility and using UI elements, usability for users with limited mobility can be improved.

[0060] FIG. 2 is a block diagram showing the configuration of an electronic device (100) according to one embodiment of the present disclosure. Referring to FIG. 2, the electronic device (100) may include at least one of a processor (111), a projection unit (112), a memory (113), a communication interface (114), an operation interface (115), an input / output interface (116), a speaker (117), a microphone (118), a power supply unit (119), a driving unit (120), or a sensor (121). Meanwhile, the configuration shown in FIG. 2 is merely an example of various embodiments, and some configurations may be omitted, and new configurations may be added.

[0061] The projection unit (112) is configured to project an image outward. According to various embodiments of the present disclosure, the projection unit (112) can be implemented using various projection methods (e.g., cathode-ray tube (CRT) method, liquid crystal display (LCD) method, digital light processing (DLP) method, laser method, etc.). For example, the CRT method is basically the same principle as a CRT monitor. The CRT method magnifies the image using a lens in front of the cathode-ray tube (CRT) to display the image on a screen. Depending on the number of cathode-ray tubes, it is divided into a single-tube type and a three-tube type, and in the case of the three-tube type, the red, green, and blue cathode-ray tubes can be implemented separately.

[0062] As another example, the LCD method displays an image by transmitting light from a light source through a liquid crystal. LCD methods are divided into single-panel and three-panel types. In the case of the three-panel type, light from a light source is separated into Red, Green, and Blue by a dichroic mirror (a mirror that reflects only specific colors of light and allows the rest to pass through), then passes through the liquid crystal, and finally converges back into one place.

[0063] As another example, the DLP method is a technique that displays images using a Digital Micromirror Device (DMD) chip. The projection unit of the DLP method may include a light source, a color wheel, a DMD chip, and a projection lens. Light emitted from the light source can acquire color as it passes through a rotating color wheel. The light passing through the color wheel is input into the DMD chip. The DMD chip contains numerous micro-mirrors and reflects the light input into it. The projection lens performs the function of magnifying the light reflected from the DMD chip to the size of an image.

[0064] As another example, the laser method includes DPSS (Diode Pumped Solid State) lasers and galvanometers. Lasers that output various colors utilize a laser formed by installing three DPSS lasers for each RGB color and superimposing their optical axes using special mirrors. The galvanometer includes mirrors and high-output motors to move the mirrors at high speeds. For example, a galvanometer can rotate the mirror at a maximum speed of 40 KHz / sec. The galvanometer is mounted according to the scanning direction; since projectors generally perform planar scanning, the galvanometer can also be positioned along the x and y axes.

[0065] Meanwhile, the projection unit (112) may include various types of light sources. For example, the projection unit (112) may include at least one light source among a lamp, an LED, and a laser.

[0066] The projection unit (112) can output images with a 4:3 aspect ratio, a 5:4 aspect ratio, or a 16:9 wide aspect ratio depending on the use of the electronic device (100) or the user's settings, and can output images with various resolutions such as WVGA (854*480), SVGA (800*600), XGA (1024*768), WXGA (1280*720), WXGA (1280*800), SXGA (1280*1024), UXGA (1600*1200), and Full HD (1920*1080) depending on the aspect ratio.

[0067] Meanwhile, the projection unit (112) can perform various functions to adjust the output image under the control of the processor (111). For example, the projection unit (112) can perform functions such as zoom, keystone, quick corner (4-corner) keystone, lens shift, etc.

[0068] Specifically, the projection unit (112) can enlarge or reduce the image depending on the distance from the screen (projection distance). That is, a zoom function can be performed depending on the distance from the screen. At this time, the zoom function may include a hardware method that adjusts the screen size by moving a lens and a software method that adjusts the screen size by cropping the image. Meanwhile, when the zoom function is performed, the focus of the image needs to be adjusted. For example, the focus adjustment method includes a manual focus method and an electric method. A manual focus method refers to a method of manually adjusting the focus, and an electric method refers to a method in which the projector automatically adjusts the focus using a motor built into it when the zoom function is performed. When performing the zoom function, the projection unit (112) may provide a digital zoom function through software and may provide an optical zoom function that performs the zoom function by moving the lens through a rotating part.

[0069] Additionally, the projection unit (112) can perform a keystone correction function. If the height is not correct for front projection, the screen may be distorted upward or downward. The keystone correction function refers to a function that corrects the distorted screen. For example, if distortion occurs in the left-right direction of the screen, it can be corrected using horizontal keystone, and if distortion occurs in the up-down direction, it can be corrected using vertical keystone. The quick corner (4 corner) keystone correction function is a function that corrects the screen when the central area of ​​the screen is normal but the corner areas are unbalanced. The lens shift function is a function that moves the screen as is when the screen is off the screen.

[0070] Meanwhile, the projection unit (112) can automatically provide zoom / keystone / focus functions by analyzing the surrounding environment and projection environment without user input. Specifically, the projection unit (112) can automatically provide zoom / keystone / focus functions based on the distance between the electronic device (100) and the screen detected through sensors (camera, distance sensor, infrared sensor, illuminance sensor, etc.), information about the space where the electronic device (100) is currently located, information about the amount of ambient light, etc.

[0071] Additionally, the projection unit (112) can provide a lighting function using a light source. In particular, the projection unit (112) can provide a lighting function by outputting a light source using an LED. According to various embodiments, the projection unit (112) may include a single LED, and according to other embodiments, the electronic device (100) may include a plurality of LEDs. Meanwhile, the projection unit (112) may output a light source using a surface-emitting LED according to an implementation example. Here, a surface-emitting LED may refer to an LED having a structure in which an optical sheet is placed on the upper side of the LED so that the light source is evenly distributed and output. Specifically, when a light source is output through the LED, the light source can be evenly distributed through the optical sheet, and the light source distributed through the optical sheet can be incident on the display panel.

[0072] Meanwhile, the projection unit (112) can provide a dimming function to the user to adjust the intensity of the light source. Specifically, when user input to adjust the intensity of the light source is received from the user through the operation interface (115) (e.g., a touch display button or a dial), the projection unit (112) can control the LED to output the intensity of the light source corresponding to the received user input.

[0073] Additionally, the projection unit (112) can provide a dimming function based on content analyzed by the processor (111) without user input. Specifically, the projection unit (112) can control an LED to output the intensity of a light source based on information about the currently provided content (e.g., content type, content brightness, etc.).

[0074] Meanwhile, the projection unit (112) can control the color temperature by the control of the processor (111). Here, the processor (111) can control the color temperature based on the content. Specifically, when the content is identified as being output, the processor (111) can obtain frame-by-frame color information of the content determined to be output. Then, the processor (111) can control the color temperature based on the obtained frame-by-frame color information. Here, the processor (111) can obtain at least one main color of the frame based on the frame-by-frame color information. Then, the processor (111) can adjust the color temperature based on at least one obtained main color. For example, the color temperature that the processor (111) can adjust can be classified as a warm type or a cold type. Here, it is assumed that the frame to be output (hereinafter output frame) contains a scene where a fire has occurred. The processor (111) can identify (or obtain) that the main color is red based on the color information included in the current output frame. And, the processor (111) can identify a color temperature corresponding to the identified primary color (red). Here, the color temperature corresponding to red may be a warm type. Meanwhile, the processor (111) may use an artificial intelligence model to obtain color information of the frame or the primary color. According to various embodiments, the artificial intelligence model may be stored in an electronic device (100) (e.g., memory (113)). According to another embodiment, the artificial intelligence model may be stored in an external server capable of communicating with the electronic device (100).

[0075] Meanwhile, the projection unit (112) may include a long-focus projection lens and an ultra-short-focus projection lens. The long-focus projection lens is a lens capable of projecting a large screen from a long distance, and can project an image from a position ranging from several meters to tens of meters away. The ultra-short-focus projection lens is a lens capable of projecting a large screen from a very short distance, and can project an image from a position ranging from several tens of centimeters away.

[0076] Additionally, the projection unit (112) may further include a switching unit capable of switching (or changing, tilting) between a long-focus projection lens and an ultra-short-focus projection lens. That is, the projector (112) can switch between a long-focus projection lens and an ultra-short-focus projection lens using the switching unit under the control of the processor (111).

[0077] The memory (113) may be implemented as internal memory such as ROM (e.g., EEPROM (electrically erasable programmable read-only memory)) or RAM included in the processor (111), or as memory separate from the processor (111). In this case, the memory (113) may be implemented as a memory embedded in the electronic device (100) or as a memory that can be attached to the electronic device (100) depending on the purpose of data storage. For example, data for operating the electronic device (100) may be stored in memory embedded in the electronic device (100), and data for the expansion function of the electronic device (100) may be stored in memory that can be attached to the electronic device (100).

[0078] Meanwhile, the memory embedded in the electronic device (100) may be implemented as at least one of volatile memory (e.g., DRAM (dynamic RAM), SRAM (static RAM), or SDRAM (synchronous dynamic RAM), non-volatile memory (e.g., OTPROM (one time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD), and the memory that is detachable from the electronic device (100) may be implemented in the form of a memory card (e.g., CF (compact flash), SD (secure digital), Micro-SD (micro secure digital), Mini-SD (mini secure digital), xD (extreme digital), MMC (multi-media card), etc.), or external memory that can be connected to a USB port (e.g., USB memory).

[0079] Memory (113) may store at least one instruction regarding the electronic device (100). Additionally, an operating system (O / S) for operating the electronic device (100) may be stored in memory (113). Furthermore, various software programs or applications for operating the electronic device (100) may be stored in memory (113) according to various embodiments of the present disclosure. Additionally, memory (113) may include semiconductor memory such as flash memory or magnetic storage media such as a hard disk.

[0080] Specifically, various software modules for operating an electronic device (100) according to various embodiments of the present disclosure may be stored in the memory (113), and the processor (111) may control the operation of the electronic device (100) by executing the various software modules stored in the memory (113). That is, the memory (113) is accessed by the processor (111), and reading / writing / modifying / deleting / updating of data by the processor (111) may be performed.

[0081] In particular, the memory (113) can store information about a map of the house where the electronic device (100) is located. In this case, the map may include a drawing that represents the planar structure of the house reduced by a certain ratio and indicated by a predefined symbol. For example, the map may include a drawing that represents the planar structure of the house interior with lines. However, it is not limited thereto, and the map may also include the locations of major objects inside the house.

[0082] Meanwhile, a map according to one embodiment of the present disclosure may be a two-dimensional map, but this is merely one embodiment and may be a three-dimensional map.

[0083] Meanwhile, in the present disclosure, the term memory (113) may be used to include a storage unit, a ROM (not shown), a RAM (not shown) within a processor (111), or a memory card (not shown) (e.g., a micro SD card, a memory stick) mounted in an electronic device (100).

[0084] The communication interface (114) is a configuration that communicates with various types of external devices according to various types of communication methods. The communication interface (114) may include a wireless communication module or a wired communication module. Here, each communication module may be implemented in the form of at least one hardware chip.

[0085] A wireless communication module may be a module that communicates wirelessly with an external device. For example, a wireless communication module may include at least one module among a Wi-Fi module, a Bluetooth module, an infrared communication module, or other communication modules.

[0086] Wi-Fi modules and Bluetooth modules can perform communication using Wi-Fi and Bluetooth methods, respectively. When using a Wi-Fi module or a Bluetooth module, various connection information, such as the SSID (service set identifier) ​​and session key, is transmitted and received first; after establishing a communication connection using this information, various types of information can be transmitted and received.

[0087] The infrared communication module performs communication according to infrared communication (IrDA, Infrared Data Association) technology, which uses infrared rays located between visible light and millimeter waves to wirelessly transmit data over short distances.

[0088] Other communication modules may include at least one communication chip that performs communication according to various wireless communication standards such as Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), and 5G (5th Generation), in addition to the communication method described above.

[0089] A wired communication module may be a module that communicates with an external device via a wire. For example, a wired communication module may include at least one of a Local Area Network (LAN) module, an Ethernet module, a pair cable, a coaxial cable, a fiber optic cable, or an Ultra Wide-Band (UWB) module.

[0090] In particular, the communication interface (114) can communicate with an external device (200). The communication interface (114) can receive information about the external device (200) from the external device (200) (e.g., information about the current location of the external device (200), information about the device equipped in the external device (200), etc.). The communication interface (114) can transmit path information to the external device (200) and can transmit information about the multi-projector system.

[0091] The operation interface (115) may include various types of input devices. For example, the operation interface (115) may include a physical button. In this case, the physical button may include a function key, a directional key (e.g., a four-way key), or a dial button. According to various embodiments, the physical button may be implemented as a plurality of keys. According to another embodiment, the physical button may be implemented as a single key. Here, when the physical button is implemented as a single key, the electronic device (100) may receive user input in which a single key is pressed for a threshold time or longer. When user input in which a single key is pressed for a threshold time or longer is received, the processor (111) may perform a function corresponding to the user input. For example, the processor (111) may provide a lighting function based on the user input.

[0092] Additionally, the operation interface (115) can receive user input using a non-contact method. When receiving user input through a contact method, physical force must be transmitted to the electronic device (100). Therefore, a method for controlling the electronic device (100) regardless of physical force may be required. Specifically, the operation interface (115) can receive user gestures and perform actions corresponding to the received user gestures. Here, the operation interface (115) can receive user gestures through a sensor (e.g., an image sensor or an infrared sensor).

[0093] Additionally, the operation interface (115) can receive user input using a touch method. For example, the operation interface (115) can receive user input through a touch sensor. Depending on various embodiments, the touch method may be implemented in a non-contact manner. For example, the touch sensor may determine whether the user's body has approached within a threshold distance. Here, the touch sensor may identify user input even when the user does not touch the touch sensor. Meanwhile, according to another embodiment, the touch sensor may identify user input when the user touches the touch sensor.

[0094] Meanwhile, the electronic device (100) may receive user input in various ways other than the operation interface (115) described above. In various embodiments, the electronic device (100) may receive user input through an external remote control device. Here, the external remote control device may be a remote control device corresponding to the electronic device (100) (e.g., a dedicated control device of the electronic device (100)) or a user's portable communication device (e.g., a smartphone or a wearable device). Here, the user's portable communication device may store an application for controlling the electronic device (100). The portable communication device may acquire user input through the stored application and transmit the acquired user input to the electronic device (100). The electronic device (100) may receive user input from the portable communication device and perform an operation corresponding to the user's control command.

[0095] Meanwhile, the electronic device (100) can receive user input using voice recognition. According to various embodiments, the electronic device (100) can receive user voice through a microphone included in the electronic device (100). According to another embodiment, the electronic device (100) can receive user voice from a microphone or an external device. Specifically, the external device can acquire user voice through the microphone of the external device and transmit the acquired user voice to the electronic device (100). The user voice transmitted from the external device may be audio data or digital data converted from audio data (e.g., audio data converted into the frequency domain). Here, the electronic device (100) can perform an operation corresponding to the received user voice. Specifically, the electronic device (100) can receive audio data corresponding to the user voice through the microphone. And, the electronic device (100) can convert the received audio data into digital data. Additionally, the electronic device (100) can convert the converted digital data into text data using a Speech To Text (STT) function. Depending on various embodiments, the Speech To Text (STT) function may be performed directly by the electronic device (100), and depending on other embodiments, the Speech To Text (STT) function may be performed by an external server. The electronic device (100) can transmit the digital data to an external server. The external server can convert the digital data into text data and obtain control command data based on the converted text data. The external server can transmit the control command data (which may also include text data) to the electronic device (100). The electronic device (100) can perform an action corresponding to the user's voice based on the obtained control command data.

[0096] Meanwhile, the electronic device (100) may provide a voice recognition function using one assistant (or an artificial intelligence assistant, e.g., Bixby™), but this is merely one of various embodiments and may provide a voice recognition function through multiple assistants. At this time, the electronic device (100) may provide a voice recognition function by selecting one of the multiple assistants based on a trigger word corresponding to the assistant or a specific key on the remote control.

[0097] Meanwhile, the electronic device (100) can receive user input using screen interaction. Screen interaction refers to a function in which the electronic device (100) identifies whether a predetermined event occurs through an image projected onto a screen (or projection surface) and obtains user input based on the predetermined event. Here, the predetermined event may refer to an event in which a predetermined object is identified at a specific location (e.g., a location where a UI for receiving user input is projected). Here, the predetermined object may include at least one of a part of the user's body (e.g., fingers, hands, feet, etc.), a pointer, or a laser pointer. When the electronic device (100) identifies a predetermined object at a location corresponding to the projected UI, it may identify that user input for selecting the projected UI has been received. For example, the electronic device (100) may project a guide image to display the UI on the screen. Then, the electronic device (100) may identify whether the user selects the projected UI. Specifically, the electronic device (100) can identify that a user has selected the projected UI when a predetermined event is identified at the location of the projected UI. Here, the projected UI may include at least one item. Here, the electronic device (100) can perform spatial analysis to identify whether the predetermined event is at the location of the projected UI. Here, the electronic device (100) can perform spatial analysis through a sensor (e.g., image sensor, infrared sensor, depth camera, distance sensor, etc.). By performing spatial analysis, the electronic device (100) can identify whether the predetermined event occurs at a specific location (the location where the UI is projected). And, if it is identified that the predetermined event occurs at a specific location (the location where the UI is projected), the electronic device (100) can identify that user input for selecting the UI corresponding to the specific location has been received.

[0098] In particular, the operation interface (115) can receive user commands for operating a multi-projection function. In one embodiment, a touch / jog icon for inputting user commands may be provided through a display. And, when a touch / jog icon is selected through a touch display or the like among the operation interfaces (115), the electronic device (100) can receive user commands (e.g., user commands for operating a multi-projection function).

[0099] The input / output interface (116) is configured to input or output at least one of an audio signal and an image signal. The input / output interface (116) can receive at least one of the audio and image signals from an external device and can output control commands to the external device.

[0100] Depending on the implementation example, the input / output interface (116) may be implemented as an interface that inputs and outputs only audio signals and an interface that inputs and outputs only image signals, or as a single interface that inputs and outputs both audio signals and image signals.

[0101] Meanwhile, in various embodiments of the present disclosure, the input / output interface (116) may be implemented as at least one wired input / output interface among HDMI (High Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), USB C-type, DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB (D-subminiature), and DVI (Digital Visual Interface). According to various embodiments, the wired input / output interface may be implemented as an interface that inputs and outputs only audio signals and an interface that inputs and outputs only image signals, or as a single interface that inputs and outputs both audio signals and image signals.

[0102] Additionally, the electronic device (100) can receive data through a wired input / output interface, but this is merely an example of various embodiments, and it can also receive power through a wired input / output interface. For example, the electronic device (100) can receive power from an external battery via a USB C-type or from an outlet via a power adapter. As another example, the electronic device (100) can receive power from an external device (e.g., a laptop or a monitor) via DP.

[0103] Meanwhile, the audio signal may be implemented to be received through a wired input / output interface, and the image signal to be received through a wireless input / output interface (or communication interface). Alternatively, the audio signal may be implemented to be received through a wireless input / output interface (or communication interface), and the image signal to be received through a wired input / output interface.

[0104] The speaker (117) is configured to output an audio signal. In particular, the speaker (117) may include an audio output mixer, an audio signal processor, and an audio output module. The audio output mixer may synthesize multiple audio signals to be output into at least one audio signal. For example, the audio output mixer may synthesize an analog audio signal and another analog audio signal (e.g., an analog audio signal received from an external source) into at least one analog audio signal. The audio output module may include a speaker or an output terminal. According to various embodiments, the audio output module may include multiple speakers, in which case the audio output module may be placed inside the main body, and the sound radiated by covering at least a part of the diaphragm of the audio output module may pass through a waveguide and be transmitted to the outside of the main body. The audio output module may include multiple audio output units, and by symmetrically arranging the multiple audio output units on the exterior of the main body, sound may be radiated in all directions, that is, in all 360 degrees.

[0105] The microphone (118) is a component for receiving user voice or other sounds and converting them into audio data. The microphone (118) can receive the user's voice when active. For example, the microphone (118) may be formed integrally on the upper side, front side, or side side of the electronic device (100). The microphone (118) may include various components such as a microphone for collecting analog user voice, an amplifier circuit for amplifying the collected user voice, an A / D conversion circuit for sampling the amplified user voice and converting it into a digital signal, and a filter circuit for removing noise components from the converted digital signal.

[0106] The power supply unit (119) can receive power from an external source and supply power to various components of the electronic device (100). The power supply unit (119) according to various embodiments of the present disclosure can receive power through various methods. In various embodiments, the power supply unit (119) can receive power using a connector. In addition, the power supply unit (119) can receive power using a 220V DC power cord. However, it is not limited thereto, and the electronic device (100) can receive power using a USB power cord or a wireless charging method.

[0107] Additionally, the power supply unit (119) may receive power using an internal battery or an external battery. In various embodiments of the present disclosure, the power supply unit (119) may receive power through an internal battery. For example, the power supply unit (119) may charge the power of the internal battery using at least one of a 220V DC power cord, a USB power cord, and a USB C-Type power cord, and receive power through the charged internal battery. In addition, in various embodiments of the present disclosure, the power supply unit (119) may receive power through an external battery. For example, when the electronic device (100) and the external battery are connected through various wired communication methods such as a USB power cord, a USB C-Type power cord, or a socket, the power supply unit (119) may receive power through the external battery. That is, the power supply unit (119) may receive power directly from the external battery, or charge the internal battery through the external battery and receive power from the charged internal battery.

[0108] The power supply unit (119) according to the present disclosure can receive power using at least one of the plurality of power supply methods described above.

[0109] Meanwhile, regarding power consumption, the electronic device (100) may have power consumption less than or equal to a preset value (e.g., 43W) due to reasons such as the socket type and other standards. At this time, the electronic device (100) may vary power consumption to reduce power consumption when using a battery. That is, the electronic device (100) may vary power consumption based on the power supply method and power usage.

[0110] The driving unit (120) can drive at least one hardware component included in the electronic device (100). The driving unit (120) can generate physical force and transmit it to at least one hardware component included in the electronic device (100). Here, the driving unit (120) can generate driving power for a movement operation of the hardware component included in the electronic device (100) (e.g., movement of the electronic device (100)).

[0111] In particular, the driving unit (120) can move the position of the electronic device (100). Here, the driving unit (120) can control the moving member (109) to move the electronic device (100). For example, the driving unit (120) can control the moving member (109) using a motor and a wheel. Meanwhile, according to the present disclosure, the driving unit (120) may be referred to by various terms such as a driving unit, a driving unit, a moving unit, etc.

[0112] The sensor (121) may include at least one sensor. Specifically, the sensor (121) may include at least one of a tilt sensor that senses the tilt of the electronic device (100) and an image sensor that captures an image. Here, the tilt sensor may be an accelerometer or a gyroscope, and the image sensor may refer to a camera or a depth camera. Meanwhile, the tilt sensor may be described as a motion sensor. In addition, the sensor (121) may include various sensors other than the tilt sensor or the image sensor. For example, the sensor (121) may include an illuminance sensor or a distance sensor. The distance sensor may be a Time of Flight (ToF). In addition, the sensor (121) may include a LiDAR sensor.

[0113] In particular, the sensor (121) can obtain a sensing value to obtain information about whether an external device (200) is located within a detection range using at least one sensor. Additionally, the sensor (121) can obtain obstacle information within a detection range using at least one sensor.

[0114] Additionally, the sensor (121) may include at least one camera for obtaining the distance from the user. In one or more embodiments, the camera may be a depth camera. Here, the depth camera is a camera that measures depth (depth, distance) information with respect to an object (e.g., the user). While a general RGB camera provides only color information, a depth camera can obtain a depth map containing distance values ​​for each pixel. In one or more embodiments, the camera may be implemented as a plurality of cameras. Specifically, depth information can be obtained by triangulation by arranging a plurality of cameras at a constant interval (baseline). Furthermore, the electronic device (100) can find the coordinates of the same object and calculate the distance using a stereo matching technique. In particular, the electronic device (100) can find the same object from a plurality of images captured by a plurality of cameras and calculate the pixel difference (disparity) to obtain a disparity map. Here, the larger the disparity value included in the disparity map, the closer the distance, and the smaller the disparity value, the farther the distance. However, this is merely one embodiment, and the distance between the electronic device (100) and the user can be obtained in various ways.

[0115] Meanwhile, the electronic device (100) can control lighting functions by linking with an external device. Specifically, the electronic device (100) can receive lighting information from an external device. Here, the lighting information may include at least one of brightness information or color temperature information set by the external device. Here, the external device may refer to a device connected to the same network as the electronic device (100) (e.g., an IoT device included in the same home / company network) or a device that is not on the same network as the electronic device (100) but can communicate with the electronic device (100) (e.g., a remote control server). For example, assume that an external lighting device (IoT device) included in the same network as the electronic device (100) is outputting red light at a brightness of 50. The external lighting device (IoT device) can transmit lighting information (e.g., information indicating that it is outputting red light at a brightness of 50) to the electronic device (100) directly or indirectly. Here, the electronic device (100) can control the output of a light source based on lighting information received from an external lighting device. For example, if the lighting information received from the external lighting device includes information for outputting red light at a brightness of 50, the electronic device (100) can output red light at a brightness of 50.

[0116] Meanwhile, the electronic device (100) can control the lighting function based on biometric information. Specifically, the processor (111) can acquire the user's biometric information. Here, the biometric information may include at least one of the user's body temperature, heart rate, blood pressure, respiration, and electrocardiogram. Here, the biometric information may include various information in addition to the information described above. As an example, the electronic device (100) may include a sensor for measuring biometric information. The processor (111) can acquire the user's biometric information through the sensor and control the output of the light source based on the acquired biometric information. As another example, the processor (111) can receive biometric information from an external device through an input / output interface (116). Here, the external device may refer to the user's portable communication device (e.g., a smartphone or a wearable device). The processor (111) can acquire the user's biometric information from the external device and control the output of the light source based on the acquired biometric information. Meanwhile, according to an embodiment, the electronic device (100) can identify whether the user is sleeping, and if the user is identified as being asleep (or preparing to sleep), the processor (111) can control the output of the light source based on the user's biometric information.

[0117] Meanwhile, an electronic device (100) according to various embodiments of the present disclosure can provide various smart functions.

[0118] Specifically, the electronic device (100) is connected to a mobile terminal device for controlling the electronic device (100), and the screen output from the electronic device (100) can be controlled through user input received from the mobile terminal device. For example, the mobile terminal device may be implemented as a smartphone including a touch display, and the electronic device (100) receives and outputs screen data provided by the mobile terminal device from the mobile terminal device, and the screen output from the electronic device (100) can be controlled according to user input received from the mobile terminal device.

[0119] The electronic device (100) can connect with a mobile terminal device through various communication methods such as Miracast, Airplay, wireless DEX, and Remote PC, and share content or music provided by the mobile terminal device.

[0120] Additionally, the mobile terminal device and the electronic device (100) can be connected in various connection methods. In various embodiments, the mobile terminal device may search for the electronic device (100) to perform a wireless connection, or the electronic device (100) may search for the mobile terminal device to perform a wireless connection. Additionally, the electronic device (100) can output content provided by the mobile terminal device.

[0121] In various embodiments, when a specific content or music is being output from a mobile terminal device, the mobile terminal device is placed near an electronic device (100), and a preset gesture is detected through the display of the mobile terminal device (e.g., motion tap view), the electronic device (100) can output the content or music being output from the mobile terminal device.

[0122] In various embodiments, when specific content or music is being output from a mobile terminal device, if the mobile terminal device comes closer to the electronic device (100) at a distance less than a preset distance (e.g., non-contact tap view) or if the mobile terminal device comes into contact with the electronic device (100) twice at short intervals (e.g., contact tap view), the electronic device (100) can output the content or music being output from the mobile terminal device.

[0123] In the above-described embodiment, it was explained that a screen identical to the screen provided by the mobile terminal device is provided by the electronic device (100), but the present disclosure is not limited thereto. That is, when a connection is established between the mobile terminal device and the electronic device (100), a first screen provided by the mobile terminal device may be output at the mobile terminal device, and a second screen provided by the mobile terminal device that is different from the first screen may be output at the electronic device (100). For example, the first screen may be a screen provided by a first application installed on the mobile terminal device, and the second screen may be a screen provided by a second application installed on the mobile terminal device. For example, the first screen and the second screen may be different screens provided by a single application installed on the mobile terminal device. In addition, for example, the first screen may be a screen that includes a UI in the form of a remote control for controlling the second screen.

[0124] The electronic device (100) according to the present disclosure may output a standby screen. For example, the electronic device (100) may output a standby screen when the electronic device (100) is not connected to an external device or when there is no input received from an external device for a preset period of time. The conditions for the electronic device (100) to output a standby screen are not limited to the above-described examples, and a standby screen may be output under various conditions.

[0125] The electronic device (100) may output a standby screen in the form of a blue screen, but the present disclosure is not limited thereto. For example, the electronic device (100) may obtain an unstructured object by extracting only the shape of a specific object from data received from an external device, and may output a standby screen including the obtained unstructured object.

[0126] Meanwhile, the electronic device (100) may further include a display (not shown).

[0127] The display (not shown) can be implemented as various types of displays, such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, or a PDP (Plasma Display Panel). The display (not shown) may also include a driving circuit, a backlight unit, etc., which can be implemented in forms such as an a-si TFT (amorphous silicon thin film transistor), an LTPS (low temperature poly silicon) TFT, or an OTFT (organic TFT). Meanwhile, the display (not shown) can be implemented as a touch screen combined with a touch sensor, a flexible display, a 3D display, or a three-dimensional display. Furthermore, according to various embodiments of the present disclosure, the display (not shown) may include not only a display panel that outputs an image but also a bezel that houses the display panel. In particular, according to various embodiments of the present disclosure, the bezel may include a touch sensor (not shown) for detecting user interaction.

[0128] Meanwhile, the electronic device (100) may further include a shutter part (not shown).

[0129] The shutter section (not shown) may include at least one of a shutter, a fixed member, a rail, or a body.

[0130] Here, the shutter can block light output from the projection unit (112). Here, the fixed member can fix the position of the shutter. Here, the rail may be a path for moving the shutter and the fixed member. Here, the body may be a configuration including the shutter and the fixed member.

[0131] The processor (111) may be implemented as a digital signal processor (DSP) that processes digital signals, a microprocessor, or a time controller (TCON). However, it is not limited thereto, and may include or be defined by one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a graphics-processing unit (GPU), a communication processor (CP), or an ARM (advanced reduced instruction set computer (RISC) machine) processor. Additionally, the processor (111) may be implemented as a System on Chip (SoC) or large scale integration (LSI) with a built-in processing algorithm, or may be implemented in the form of a Field Programmable Gate Array (FPGA). Furthermore, the processor (111) can perform various functions by executing computer executable instructions stored in memory (113).

[0132] In particular, the processor (111) can project a UI including multiple UI elements through the projection unit (112) by executing at least one command stored in the memory (113). The processor (111) can acquire an image related to the user's movement corresponding to the selection of one of the multiple UI elements through a camera. The processor (111) can acquire information corresponding to the distance between the electronic device (100) and multiple feature regions related to the user's foot from the image. The processor (111) can select one of the multiple UI elements based on the information corresponding to the distance.

[0133] In one or more embodiments, the processor (111) may acquire (or extract) a plurality of feature regions related to the user's feet when it is identified that the user is in a wheelchair. Here, the plurality of feature regions may include a first feature region corresponding to a first frame of the wheelchair and a right footrest corresponding to the user's right foot, and a second feature region corresponding to a second frame of the wheelchair and a left footrest corresponding to the user's left foot.

[0134] In one or more embodiments, the processor (111) may acquire (or identify) information regarding the direction of rotation of a user based on information corresponding to the distance between a plurality of feature regions and an electronic device (100), and may select a UI element corresponding to the direction of rotation of a user among a plurality of UI elements based on the information regarding the direction of rotation of a user.

[0135] In one or more embodiments, the processor (111) identifies that the user has rotated counterclockwise if the first distance between the first feature area and the electronic device (100) is shorter than the second distance between the second feature area and the electronic device (100), and identifies that the user has rotated clockwise if the first distance between the first feature area and the electronic device (100) is longer than the second distance between the second feature area and the electronic device (100).

[0136] In one or more embodiments, the processor (111) obtains a correction value based on the difference between the first distance between the first feature area and the electronic device (100) and the second distance between the second feature area and the electronic device (100) before the user moves, and can select one of a plurality of UI elements based on information corresponding to the correction value and the distance.

[0137] In one or more embodiments, the processor (111) can perform keystone correction to make the UI parallel to the user based on an image captured through a camera before the user moves, if the UI and the user do not correspond to a parallel state.

[0138] In one or more embodiments, the processor (111) can drive the electronic device (100) so that the user and the UI correspond to a parallel state based on an image captured through a camera before the user moves, if the UI and the user do not correspond to a parallel state.

[0139] In one or more embodiments, when a first event corresponding to UI projection is identified based on a first user who is not in a wheelchair among a plurality of users, the processor (111) controls the drive unit (120) to drive to an area corresponding to the first user based on the first event, and while projecting the UI to the area corresponding to the first user, can obtain information corresponding to the first user's dominant foot based on the first user's foot movement.

[0140] Additionally, the processor (111) can obtain information corresponding to the primary foot of the first user, and when a second event corresponding to UI projection based on the first user is detected, control the driving unit (120) to drive to an area corresponding to the first user based on the second event, and project a modified UI in which the arrangement of UI elements included in the UI is changed based on the information regarding the primary foot of the first user.

[0141] FIGS. 3a and 3b are drawings illustrating the external appearance of an electronic device (100) according to various embodiments of the present disclosure. As shown in FIG. 3a, the electronic device (100) may include a sensor (121) (e.g., a camera) and a projection unit (112) in a head area. Here, the head area refers to an area located at the top of the electronic device (100) and may be referred to by various terms such as top area, top area, sensing area, projection area, etc. Furthermore, the head area of ​​the electronic device (100) may be rotated to adjust the projection angle or to sense information about the surroundings of the electronic device (100). At this time, the head area of ​​the electronic device (100) may be rotated by a rotation unit (310) as shown in FIG. 3. In one or more embodiments, the head area of ​​the electronic device (100) may perform a yaw rotation that rotates in the left and right directions. Additionally, the head region of the electronic device (100) can perform pitch rotation, which rotates in the up and down direction. Additionally, the head region of the electronic device (100) can perform roll rotation, which rotates in the clockwise / counterclockwise direction. However, although the head region of the electronic device (100) can rotate in all three axes, this is merely one embodiment, and it is obvious that it can rotate in only some of the three axes.

[0142] Additionally, a driving unit (120) may be included in the lower region of the electronic device (100). The driving unit (120) can move the electronic device (100) by driving a moving member (i.e., a wheel).

[0143] That is, the electronic device (100) can adjust the projection angle of the electronic device (100) through the rotating part (310) and adjust the projection position of the electronic device (100) through the driving part (120).

[0144] FIG. 3b is a drawing illustrating the external appearance of an electronic device according to an embodiment of the present disclosure. Referring to the embodiment of FIG. 3, the electronic device (100) may include a moving member (109). The moving member (109) may refer to a member for moving from a first position to a second position in a space where the electronic device (100) is placed. The electronic device (100) may control the moving member (109) so that the electronic device (100) moves using a force generated by a driving unit (120). At this time, the moving member (109) may include a motor or a wheel. Additionally, as shown in FIG. 3b, the electronic device (100) may include a projection unit (112) on a part of the main body.

[0145]

[0146] FIG. 4 is a flowchart illustrating an embodiment of selecting a UI element based on a user's foot movement according to one embodiment of the present disclosure.

[0147] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0148] According to one embodiment, operations S410 to S460 may be understood to be performed in a processor (e.g., processor (111) of FIG. 2) of an electronic device (e.g., electronic device (100) of FIG. 2).

[0149] In one or more embodiments, the electronic device (100) can detect an event for projecting a UI to a user (S410). Here, the event for projecting the UI may be an event in which the user inputs a UI projection command, but is not limited thereto, and may be various events such as an event reaching a preset time or an event in which the user approaches the electronic device (100).

[0150] The electronic device (100) can project a UI including multiple UI elements through a projection unit (112) (S420). Specifically, the electronic device (100) can identify the user's location. In one or more embodiments, the electronic device (100) can identify the user's location using a sensor and a camera, but is not limited thereto, and can identify the user's location by receiving location information of the user through an external device. Furthermore, the electronic device (100) can move around the user (or an area corresponding to the user) through a driving unit (120) based on the user's location. The electronic device (100) can identify a projection surface around the user to project the UI. Here, the projection surface is an area for the electronic device (100) to project an image, and may be, for example, a floor, but this is merely one embodiment, and may be a wall or a screen of a preset color (e.g., white). Then, the electronic device (100) can project a UI including multiple UI elements onto the identified projection surface. Here, the UI may include various UI elements, such as UI elements containing information to be provided to the user, UI elements for controlling the electronic device (100), etc. Here, the UI elements may be implemented as various elements such as icons, items, thumbnail images, indicators, etc.

[0151] The electronic device (100) can acquire an image through a camera (S430). Here, the image is an image including a user and can be used to acquire information about the movement of a user selecting one of a plurality of UI elements. Here, the camera can be implemented as a depth camera, but is not limited thereto, and can be implemented as a plurality of cameras. Here, the image can be a depth image, but is not limited thereto, and can be an RGB image.

[0152] The electronic device (100) can extract multiple feature regions related to the user's foot (S440). Specifically, the electronic device (100) can identify whether the user is a wheelchair user by analyzing a captured image. If the user is identified as a wheelchair user, the electronic device (100) can extract multiple feature regions related to the user's foot.

[0153] In one or more embodiments, the electronic device (100) may acquire an image (500) including a user, as shown on the left side of FIG. 5. Here, the image (500) may include a user in a wheelchair. The electronic device (100) may extract from the image (500) a first feature region (510-1) corresponding to the first frame of the wheelchair and the right footrest corresponding to the user's right foot, and a second feature region (510-2) corresponding to the second frame of the wheelchair and the left footrest corresponding to the user's left foot. The electronic device (100) may acquire a feature region image including the first and second feature regions (510-1, 510-2), as shown on the right side of FIG. 5. Meanwhile, the feature region is an area related to the user's movement (particularly, foot movement) and may be referred to by various terms such as a region of interest, a target region, etc.

[0154] Here, the electronic device (100) can identify both feet of the user through the acquired image and extract multiple feature regions located around both feet of the user. However, this is merely one embodiment, and the electronic device (100) can extract multiple feature regions directly. For example, if the user is missing at least one of both feet or has a cast, the user's feet cannot be identified, so the electronic device (100) can extract multiple feature regions directly without identifying the user's feet.

[0155] However, identifying a user in a wheelchair is merely one example, and other users with limited mobility (e.g., a user holding crutches or a person with a lower limb amputation) can be identified. If the user is a general user, the electronic device (100) can extract the user's feet or hands as feature regions.

[0156] The electronic device (100) can obtain information corresponding to the distance between the electronic device (100) and a plurality of feature regions related to the user's foot (S450). Specifically, the electronic device (100) can obtain information regarding the user's rotation direction based on the information corresponding to the distance between the plurality of feature regions and the electronic device.

[0157] In one or more embodiments, the electronic device (100) may obtain information regarding a first distance (D1) between the electronic device (100) and a feature area corresponding to the user (10)'s right foot and a second distance (D2) between the electronic device (100) and a feature area corresponding to the user (10)'s left foot while the UI (610) is projected, as shown in FIG. 6a, before the user moves. Here, D1 and D2 may be the same, but are not limited thereto.

[0158] Meanwhile, if D1 and D2 are not identical, the difference between D1 and D2 can be obtained as a correction value. Here, the correction value can be used later to identify the user's rotation direction after the user moves.

[0159] And, when the user moves (or rotates), the electronic device (100) can obtain information corresponding to the distance between the electronic device (100) and a plurality of feature regions related to the user's foot. Specifically, if the first distance (D1) between the first feature region and the electronic device (100) becomes shorter than the second distance (D2) between the second feature region and the electronic device (100), the electronic device (100) can identify that the user has rotated counterclockwise. Additionally, if the first distance (D1) between the first feature region and the electronic device (100) becomes longer than the second distance (D2) between the second feature region and the electronic device (100), the electronic device (100) can identify that the user has rotated clockwise. This can be expressed mathematically as follows.

[0160] [Mathematical Formula 1]

[0161] D1 - D2 > 0 : Press Right

[0162] D1 - D2 < 0 : Press Left

[0163] In one or more embodiments, as illustrated in FIG. 6b, when the user's right foot moves forward, the electronic device (100) identifies that the first distance (D1) has become shorter than the second distance (D2), so the electronic device (100) can identify that it has rotated counterclockwise.

[0164] Meanwhile, if the first distance (D1) and the second distance (D2) are not the same before the user moves, that is, if a correction value exists, the electronic device (100) can identify the direction of rotation of the user based on information corresponding to the correction value and the distance. Specifically, the electronic device (100) can identify the direction in which the user rotated by comparing the difference between the first distance (D1) and the second distance (D2) with the correction value after the user moves. For example, if the difference between the first distance (D1) and the second distance (D2) is greater than the correction value, the electronic device (100) can identify that the user rotated in a counterclockwise direction, and if the difference between the first distance (D1) and the second distance (D2) is smaller than the correction value, the electronic device (100) can identify that the user rotated in a clockwise direction.

[0165] The electronic device (100) can select one of a plurality of UI elements based on information corresponding to distance (S460). Specifically, the electronic device (100) can select a UI element corresponding to the user's rotation direction identified based on information corresponding to distance. And, the electronic device (100) can control the functions of the electronic device (100) according to the selected UI element.

[0166] In one or more embodiments, as illustrated in FIG. 6b, when the user rotates so that the user's right foot faces forward, the electronic device (100) can select a "later" UI element among a plurality of UI elements. By doing so, the electronic device (100) can execute a function corresponding to the selected UI element (e.g., a later selection function).

[0167] Meanwhile, as illustrated in FIG. 6a, the user and the projected UI (610) may be parallel to each other, but the user and the projected UI cannot always be parallel to each other. Since there is a possibility of malfunction occurring when the user and the projected UI are not parallel to each other, the electronic device (100) can perform a correction operation so that the user and the projected UI (610) are parallel to each other.

[0168] In one or more embodiments, the electronic device (100) can identify whether the UI and the user (10) are parallel based on an image captured through a camera before the user moves. As illustrated in FIG. 7a, if the UI (710) and the user (10) are not parallel, the electronic device (100) can project the UI (720) by performing keystone correction to make it parallel to the user, as illustrated in FIG. 7b. That is, the electronic device (100) can correct the UI (720) so that the user and the UI (720) are parallel to each other through keystone correction. And, as illustrated in FIG. 7c, if the user rotates counterclockwise, the electronic device (100) can identify that the direction of rotation of the user is to the right using the method described in FIG. 4. And, the electronic device (100) can select the "later" UI element located on the right among a plurality of UI elements. By doing so, the electronic device (100) can execute a function corresponding to a selected UI element.

[0169] In one or more embodiments, the electronic device (100) can identify whether the UI and the user (10) are parallel based on an image captured through a camera before the user moves. As shown in FIG. 8a, if the UI (810) and the user (10) are not parallel, the electronic device (100) can move the position of the electronic device (100) through the driving unit (120) so that the UI (810) is parallel to the user, as shown in FIG. 8b. By doing so, the electronic device (100) can project the UI (810) that is parallel to the user at the moved position. As shown in FIG. 8c, if the user rotates counterclockwise, the electronic device (100) can identify that the direction of rotation of the user is to the right using the method described in FIG. 4. Then, the electronic device (100) can select the "later" UI element located on the right among a plurality of UI elements. By doing so, the electronic device (100) can execute a function corresponding to a selected UI element.

[0170]

[0171] FIG. 9 is a flowchart illustrating a control method for an electronic device capable of providing a UI corresponding to the characteristics of a plurality of users, according to one embodiment of the present disclosure.

[0172] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0173] According to one embodiment, operations S905 to S965 may be understood to be performed in a processor (e.g., processor (111) of FIG. 2) of an electronic device (e.g., electronic device (100) of FIG. 2).

[0174] In one or more embodiments, the electronic device (100) can set a personalized foot touch action option (S905). Here, the personalized foot touch action may be an action for providing a UI for different foot touches depending on the characteristics of the user (e.g., whether the user is in a wheelchair, type of dominant foot, etc.). In one or more embodiments, the electronic device (100) can set a personalized foot touch action option through the electronic device (100) or a mobile device linked to the electronic device (100).

[0175] The electronic device (100) can detect the occurrence of an event to display a UI (S910). In one or more embodiments, the electronic device (100) can display a UI when a personalized event occurs in response to a user request. However, this is merely one embodiment, and various events may occur, such as an event reaching a preset time or an event where a user approaches the electronic device (100).

[0176] The electronic device (100) can perform a user search operation (S915). Specifically, the electronic device (100) can detect a target user based on user identification data (e.g., face image, etc.) defined within the event of the S910 operation, and control the driving unit (120) to drive to the location of the detected target user.

[0177] The electronic device (100) can assign (assign) IDs to users detected while driving (S920). That is, the electronic device (100) can identify users detected while driving and assign an ID to each of the identified users.

[0178] The electronic device (100) can search for target users (S925). In one or more embodiments, the electronic device (100) can search for target users by performing face recognition for each ID of users detected while driving.

[0179] The electronic device (100) can detect a target user and store their ID (S930). In one or more embodiments, the electronic device (100) can perform a face recognition operation to identify a user matching the user's identification data within a personalization event as a target user, and store the ID of the identified target user.

[0180] The electronic device (100) can approach the target user (S935). That is, the electronic device (100) can approach the target user within a preset distance to provide a UI to the target user.

[0181] The electronic device (100) can project a UI according to the target user (S940). Specifically, the electronic device (100) can project a UI for controlling information or controlling the electronic device (100) or other electronic devices based on the characteristics of the target user. For example, the electronic device (100) can project a UI by identifying whether the target user is a user with mobility difficulties, etc.

[0182] The electronic device (100) can identify the target user's primary foot (S945). Here, the primary foot is the foot that the user frequently uses, and the electronic device (100) can identify the target user's primary foot based on previously stored information.

[0183] The electronic device (100) can correct the UI according to the dominant foot option (S950). In one or more embodiments, when the dominant foot option is set in the electronic device (100), the electronic device (100) can correct the position of UI elements displayed in the UI according to the target user's dominant foot. If the user's dominant foot is the right foot, the electronic device (100) can move the position of the UI elements included in the UI to the right so that the user can easily select them. Or, if the user is a wheelchair user, the electronic device (100) can provide the UI using the method described above in FIGS. 4 to 8c.

[0184] The electronic device (100) can track the user's location and correct the location value (S955). That is, if the user's location has moved, the electronic device (100) can track the moved location and correct the location value. Then, the electronic device (100) can select a UI element or move the UI according to the corrected location value.

[0185] The electronic device (100) can select UI elements according to user movement (S960). In one or more embodiments, the electronic device (100) can select UI elements located on the right side when the user moves to the right.

[0186] The electronic device (100) can control the electronic device (100) according to the selected UI element (S965).

[0187] FIGS. 10a to 11b are drawings for explaining a method of correcting a UI according to user characteristics according to one embodiment of the present disclosure.

[0188] In one or more embodiments, when a first event for projecting a UI based on a first user who is not in a wheelchair among a plurality of users is detected, the electronic device (100) can control the driving unit (120) to drive toward the first user in response to the first event. And, while projecting a UI around the first user, information about the primary foot frequently used by the first user can be acquired and stored based on the foot movements of the first user. For example, if the first user selects a UI element using the left foot, the electronic device (100) can store the first user's primary foot as the left foot. However, this is merely one embodiment, and the electronic device (100) can acquire information about the user's primary foot by tracking the user's movements or actions after assigning IDs to the users.

[0189] After information regarding the primary foot of the first user is stored, when a second event for projecting a UI to the first user is detected, the electronic device (100) can control the driving unit (120) to drive toward the first user in response to the second event. Here, the electronic device (100) can project a UI (1010) including a first UI element (1010-1) and a second UI element (1010-2) around the first user, as shown in FIG. 10a.

[0190] The electronic device (100) can control the projection unit (112) to project a modified UI in which the arrangement of UI elements included in the UI is changed based on information about the first user's dominant foot. For example, the electronic device (100) can control the projection unit (112) to project a modified UI (1020) in which the positions of the first UI element (1010-1) and the second UI element (1010-2) are moved to the left, as shown in FIG. 10b, so that the UI elements (1010-1, 1010-2) can be selected using the left foot, which is the first user's dominant foot.

[0191] Meanwhile, as described above, adjusting the layout of the UI based on the user's dominant foot is merely one embodiment, and it goes without saying that the layout of the UI can also be adjusted based on the user's dominant hand.

[0192] Specifically, the electronic device (100) can store the second user's dominant hand as the right hand. After the information regarding the second user's dominant hand is stored, when a second event for projecting a UI to the second user is detected, the electronic device (100) can control the driving unit (120) to drive toward the second user in response to the second event. Here, the electronic device (100) can project a UI (1110) including a first UI element (1110-1) and a second UI element (1110-2) around the second user, as shown in FIG. 11a.

[0193] The electronic device (100) can control the projection unit (112) to project a modified UI in which the arrangement of UI elements included in the UI is changed based on information about the second user's dominant hand. For example, the electronic device (100) can control the projection unit (112) to project a modified UI (1120) in which the positions of the first UI element (1110-1) and the second UI element (1110-2) are moved to the right, as shown in FIG. 11b, so that the second user can select the UI elements (1110-1, 1110-2) using the right hand, which is the second user's dominant hand.

[0194]

[0195] Meanwhile, the method according to various embodiments of the present disclosure may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0196] A method according to various embodiments of the present disclosure may be implemented as software comprising instructions stored on a machine-readable storage medium (e.g., a computer). The machine may include an electronic device (e.g., a TV) according to the disclosed embodiments, which is a device capable of calling instructions stored from the storage medium and operating according to the called instructions.

[0197] Meanwhile, a device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory storage medium' simply means that it is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily. For example, a 'non-transitory storage medium' may include a buffer in which data is stored temporarily.

[0198] When the above instruction is executed by a processor, the processor may perform the function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or an interpreter.

[0199] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

1. In an electronic device, camera; A driving unit configured to move the above electronic device; A projection unit that outputs light from a light source to a projection surface located outside the electronic device; Memory configured to store instructions; and One or more processors configured to execute the stored instructions collectively or individually; comprising, The above-mentioned stored instructions are, A UI including a plurality of UI elements is projected onto the projection surface through the projection unit, and An image related to user movement representing a user's selection of one of the plurality of UI elements is obtained through the camera, and Information corresponding to the distance between the feature region related to the user's foot and the electronic device is obtained from the above-mentioned acquired image, and An electronic device configured to select a projected UI element among the plurality of UI elements that appears to be selected by the user due to the user's movement, based on the information obtained above.

2. In Paragraph 1, The above-mentioned stored instructions are, Determining whether the above user is seated on a wheelchair, and If the above user is identified as being seated in a wheelchair, the above user is identified as a wheelchair user, and An electronic device configured to acquire a feature region related to the user's foot while the user is identified as the wheelchair user.

3. In Paragraph 2, The above feature region is, A first frame of the wheelchair corresponding to the right foot of the wheelchair user and a first feature region corresponding to the support of the wheelchair for the right foot of the wheelchair user; and An electronic device comprising: a second frame of a wheelchair corresponding to the left foot of the wheelchair user; and a second feature region corresponding to a wheelchair support for the left foot of the wheelchair user.

4. In Paragraph 3, The above-mentioned stored instructions are, Based on the acquired information corresponding to the distance between the above feature region and the above electronic device, information regarding the direction in which the wheelchair user has rotated is acquired as the direction of rotation of the wheelchair user, and An electronic device configured to select a UI element corresponding to the user's rotation direction among the plurality of UI elements based on information regarding the rotation direction obtained above.

5. In Paragraph 4, The above-mentioned stored instructions are, If the first distance between the first feature area and the electronic device is shorter than the second distance between the second feature area and the electronic device, the user's rotation in a counterclockwise direction is identified, and An electronic device configured to identify the user's clockwise rotation when the first distance is longer than the second distance.

6. In Paragraph 3, The above-mentioned stored instructions are, A correction value is obtained based on the difference between the first distance between the first feature area and the electronic device and the second distance between the second feature area and the electronic device before the wheelchair user moves, and An electronic device configured to select the projected UI element among the plurality of UI elements based on the above-mentioned acquired correction value and the above-mentioned acquired information.

7. In Paragraph 1, The above-mentioned stored instructions are, An image of the user is acquired through the camera before the user moves, and An electronic device configured to perform keystone correction to make the projected UI parallel to the user based on the acquired image before the user moves, if the projected UI is not parallel to the user.

8. In Paragraph 1, The above-mentioned stored instructions are, An image of the user is acquired through the camera before the user moves, and An electronic device configured to drive the electronic device through the driving unit so that the user and the UI become parallel based on the acquired image before the user moves.

9. In Paragraph 1, The above-mentioned stored instructions are, When a first event corresponding to the projected UI is detected in relation to a first user who is not in a wheelchair among multiple users, the electronic device is driven to an area corresponding to the first user through the driving unit, and Projecting a UI onto an area corresponding to the first user through the projection unit, and An electronic device configured to acquire information about the primary foot of the first user based on the foot movements of the first user.

10. In Paragraph 9, The above-mentioned stored instructions are, When information corresponding to the primary foot of the first user is obtained, and a second event related to the first user and corresponding to the projected UI is detected, the electronic device is driven to an area corresponding to the first user through the driving unit, and An electronic device configured to project a modified UI in which the arrangement of the plurality of UI elements is changed based on information regarding the primary foot of the first user through the projection unit.

11. A method for controlling an electronic device comprising a camera, a driving unit configured to move the electronic device, a projection unit that outputs light from a light source to a projection surface located outside the electronic device, a memory configured to store instructions, and one or more processors configured to execute the stored instructions collectively or individually, The above control method is by the at least one processor, A step of projecting a UI including a plurality of UI elements onto the projection surface through the projection unit; A step of acquiring an image related to a user's movement indicating a user's selection of one of the plurality of UI elements through the camera; A step of obtaining information corresponding to the distance between the electronic device and a feature region related to the user's foot from the above-mentioned acquired image; and A control method comprising the step of selecting a projected UI element among the plurality of UI elements that appears to be selected by the user due to the user's movement, based on the information obtained above.

12. In Paragraph 11, The above control method is, A step of determining whether the above user is seated on a wheelchair; If the above user is identified as being on a wheelchair, the step of identifying the user as a wheelchair user; and A control method comprising the step of acquiring a feature region related to the user's foot while the user is identified as the wheelchair user.

13. In Paragraph 12, The above feature region is, A first frame of the wheelchair corresponding to the right foot of the wheelchair user and a first feature region corresponding to the support of the wheelchair for the right foot of the wheelchair user; and A control method comprising: a second frame of a wheelchair corresponding to the left foot of the wheelchair user; and a second feature region corresponding to a wheelchair support for the left foot of the wheelchair user.

14. In Paragraph 13, The above-mentioned selection step is, A step of obtaining information regarding the direction in which the wheelchair user has rotated as the direction of rotation of the wheelchair user, based on the obtained information corresponding to the distance between the above feature region and the above electronic device; and A control method comprising the step of selecting a UI element corresponding to the user's rotation direction among the plurality of UI elements based on the information regarding the rotation direction obtained above.

15. In Paragraph 14, The above identification step is, If the first distance between the first feature area and the electronic device is shorter than the second distance between the second feature area and the electronic device, the user's rotation in a counterclockwise direction is identified, and A control method for identifying clockwise rotation of the user when the first distance is longer than the second distance.

Images