Electronic device and control method therefor

Abstract

Disclosed is an electronic device. The electronic device comprises a camera, a driving unit for moving the electronic device, a memory for storing at least one instruction, and at least one processor for executing the at least one instruction, wherein the at least one processor: acquires an image corresponding to a subject by means of the camera; and controls the driving unit to drive the electronic device when, on the basis of subject information included in an image, it is identified that driving the electronic device is necessary, wherein the subject information includes at least one of a facial direction of the subject, a pose corresponding to the subject, or a subject gesture.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device that drives based on subject information and a method for controlling the same.

[0002] Thanks to advancements in electronic technology, various types of electronic devices are being used in daily life. Among these devices may be electronic devices that travel through space using a driving mechanism.

[0003] For example, there may be a projector that travels through space and projects an image onto a projection surface such as a wall.

[0004] A projector is a device that displays an image by projecting light containing an image onto an external screen or wall.

[0005] According to at least one embodiment of the present disclosure, an electronic device comprises a camera, a driving unit for moving the electronic device, a memory for storing at least one instruction, and at least one processor for executing at least one instruction. The processor may acquire an image corresponding to a subject through the camera, and if it is identified that driving of the electronic device is required based on subject information included in the image, the driving device may control the electronic device to drive. The subject information may include at least one of the subject's facial orientation, a pose corresponding to the subject, and a subject gesture.

[0006] Additionally, according to at least one embodiment of the present disclosure, the control method may include the steps of acquiring an image corresponding to a subject through an electronic device, acquiring subject information included in the image, and, if it is identified that driving of the electronic device is required based on the subject information, driving of the electronic device. The subject information may include at least one of the facial direction of the subject, a pose corresponding to the subject, and a subject gesture.

[0007] Additionally, a non-transient readable recording medium according to at least one embodiment of the present disclosure stores a program for performing a method of controlling an electronic device, comprising the steps of acquiring an image corresponding to a subject through an electronic device, acquiring subject information included in the image, and, if it is identified that driving of the electronic device is required based on the subject information, driving of the electronic device. The subject information may include at least one of the facial direction of the subject, a pose corresponding to the subject, and a subject gesture.

[0008] FIG. 1 is a diagram for schematically illustrating the operation of an electronic device according to at least one embodiment of the present disclosure.

[0009] FIG. 2 is a drawing illustrating the form of an electronic device according to at least one embodiment of the present disclosure.

[0010] FIG. 3 is a block diagram illustrating the configuration of an electronic device according to at least one embodiment of the present disclosure.

[0011] FIG. 4 is a detailed block diagram for illustrating an electronic device according to at least one embodiment of the present disclosure.

[0012] FIG. 5 is a drawing for explaining the operation of an electronic device running according to at least one embodiment of the present disclosure.

[0013] FIG. 6 is a flowchart illustrating the overall operation of an electronic device according to at least one embodiment of the present disclosure.

[0014] FIG. 7 is a flowchart illustrating the overall operation of an electronic device according to at least one embodiment of the present disclosure.

[0015] FIG. 8 is a diagram illustrating the operation of an electronic device providing gesture information according to at least one embodiment of the present disclosure.

[0016] FIG. 9a is a drawing for explaining the operation of an electronic device providing guide information according to at least one embodiment of the present disclosure.

[0017] FIG. 9b is a drawing for explaining the operation of an electronic device providing guide information according to at least one embodiment of the present disclosure.

[0018] FIG. 10 is a flowchart illustrating the operation of an electronic device running according to at least one embodiment of the present disclosure.

[0019] The terms used in the various embodiments of this Disclosure have been selected to be as widely used and general as possible, taking into account their functions within this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been selected at the applicant's discretion, and in such cases, their meanings will be described in detail in the relevant description section of this disclosure. Therefore, terms used in this disclosure should be defined not merely by their names, but based on their meanings and the overall content of this disclosure.

[0020] The various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features described in the present disclosure to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments.

[0021] In relation to the description of the drawings, similar reference numerals may be used for similar or related components.

[0022] The singular form of the noun corresponding to the item may include one or multiple items, unless the relevant context clearly indicates otherwise.

[0023] In the present disclosure, each of the phrases such as “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “at least one of A, B, or C” may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

[0024] Terms such as "first," "second," or "first" or "second" may be used simply to distinguish a component from another component and do not limit the components in other aspects (e.g., importance or order).

[0025] Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that the component may be connected to the other component directly (e.g., via a wire), wirelessly, or through a third component.

[0026] Terms such as "include" or "have" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in this disclosure, and do not preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0027] When it is said that a component is "connected," "combined," "supported," or "in contact" with another component, this includes not only cases where the components are directly connected, combined, supported, or in contact, but also cases where they are indirectly connected, combined, supported, or in contact through a third component.

[0028] When it is said that a component is located "on" another component, this includes not only cases where one component is in contact with the other, but also cases where another component exists between the two components.

[0029] The term "and / or" includes a combination of multiple related described components or any of the multiple related described components.

[0030] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Additionally, a plurality of "modules" or a plurality of "parts" may be integrated into at least one module and implemented by at least one processor, except for a "module" or "part" that needs to be implemented in specific hardware.

[0031] Meanwhile, various elements and areas in the drawings are depicted schematically. Accordingly, the technical concept of the present disclosure is not limited by the relative sizes or spacing depicted in the attached drawings.

[0032] In the present disclosure, the term "user" may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0033] An embodiment of the present disclosure will be described in more detail below with reference to the attached drawings.

[0034] FIG. 1 is a diagram for schematically illustrating the operation of an electronic device according to at least one embodiment of the present disclosure.

[0035] Referring to FIG. 1, the electronic device (100) can identify the gesture of a subject (10) while traveling through space. Here, "space" may refer to a space in which the electronic device (100) can travel.

[0036] For example, when a subject (10) makes a gesture with its hand, the electronic device (100) acquires a captured image of the subject (10), and if it is identified that driving of the electronic device (100) is required based on the subject information included in the image, it can drive through the space.

[0037] Here, "driving" may include the action of an object moving using power.

[0038] According to embodiments of the present disclosure, driving may include the movement of an object moving in any direction through power. Alternatively, driving may include the movement of an object moving along a preset track through power.

[0039] For example, driving may include the action of changing the position of a mobile projector during the process of projecting an image.

[0040] Cases identified here as requiring driving may include cases where the face direction of the subject (10) in the captured image does not face in a straight line with the camera scanning direction of the electronic device (100).

[0041] For example, if the camera scanning direction of the electronic device (100) deviates from a straight line with the direction of the subject's face (10), an error may occur in misidentifying the gesture of the subject (10). Therefore, the electronic device (100) may drive to change the camera scanning direction of the electronic device (100) based on the direction of the subject's face (10) in order to accurately identify the gesture of the subject (10).

[0042] In FIG. 1, the electronic device (100) is shown in the form of a portable projector, but is not limited thereto, and the electronic device (100) can be implemented as various types of electronic devices such as a fixed, mounted, ceiling-mounted, portable projector, or an electronic device that can be carried by a user.

[0043] Here, a projector refers to a device that displays an image by projecting light containing an image onto an external screen or wall, and a portable projector refers to a projector that can be moved using wheels, motors, etc.

[0044] In addition, the projector included in the electronic device (100) illustrated in FIG. 1 can be implemented as various types of projectors, such as a CRT (Cathode-Ray Tube) projector, an LCD (Liquid Crystal Display) projector, a DLP (Digital Light Processing) projector, an LED (Light Emitting Diode) projector, or an LCOS (Liquid Crystal on Silicon) projector, depending on the method of projecting images. However, it is not limited to these and can be implemented as various devices equipped with image projection functions.

[0045] Additionally, although the electronic device (100) illustrated in FIG. 1 is shown in the form of a projector, the electronic device (100) can be implemented as a device configured to be detachably attached to an external projection device that performs the function of projecting an image.

[0046] For example, the electronic device (100) may perform an image projection function in the form of being attached to an external projection device that performs the function of projecting an image.

[0047] FIG. 2 is a drawing illustrating the form of an electronic device according to at least one embodiment of the present disclosure.

[0048] Referring to FIG. 2, a movable electronic device (100) is illustrated. This is merely an example of a disclosed form and may have a shape different from the illustrated form.

[0049] The electronic device (100) can acquire an image corresponding to a subject through a camera (110).

[0050] An image according to one example may include a captured image acquired using a sensor (e.g., a camera) equipped in an electronic device, an input image received from an external device through a communication unit, a graphic image generated by the electronic device, etc.

[0051] Images according to one example may include, depending on the aspect ratio, horizontal images where the width is longer than the height (e.g., landscape images, horizontal images), vertical images where the height is longer than the width (e.g., portrait images, vertical images), etc. For example, horizontal images may include images with a 16:9 aspect ratio, and vertical images may include images with a 9:16 aspect ratio. Specific numbers are examples for convenience of explanation and are not limited thereto.

[0052] An image according to one example may include various resolutions depending on the number of pixels constituting the image (the product of the number of pixels in the horizontal direction and the number of pixels in the vertical direction). For example, depending on the resolution, the image may include high-resolution images (FHD (1920Y1080), 8K (7680Y4320), etc.) and low-resolution images (640Y480, etc.).

[0053] The electronic device (100) can travel through space based on subject information included in the image through the driving unit (120).

[0054] Here, "subject information" may mean various information about the subject and may include at least one of the subject's facial orientation, a pose corresponding to the subject, and a subject gesture.

[0055] A shooting according to one example of the present disclosure may include the operation of an electronic device that controls a camera (e.g., a camera including an image sensor and a lens) equipped in an electronic device to convert an optical image formed through a lens into an electrical signal and acquire an image.

[0056] For example, one or more processors may control a camera equipped in an electronic device to photograph the surroundings of the electronic device and acquire an image (e.g., a captured image) containing one or more frames. Here, the image may include a live-view image.

[0057] The electronic device (100) includes a drive unit (120) for movement (e.g., a motor, a wheel) and can perform movement using the provided drive unit. Meanwhile, although the illustrated example shows and describes the electronic device (100) being moved using a wheel, it is possible to use other means in the form of a caterpillar when implementing it, and if the electronic device (100) is implemented as a drone, it is possible to equip a propeller instead of a wheel.

[0058] Meanwhile, in the illustrated example, the electronic device (100) is shown as having a direct drive unit (120), but the drive unit (120) may be a separate device. For example, the electronic device (100) may be combined with a movable device such as a robot vacuum cleaner, and may be mounted on the robot vacuum cleaner to control the movement of the robot vacuum cleaner.

[0059] Meanwhile, the form of the illustrated electronic device is merely an example and can be implemented in various forms. Specific configurations constituting the electronic device (100) will be described later with reference to FIG. 3.

[0060] FIG. 3 is a block diagram illustrating the configuration of an electronic device according to at least one embodiment of the present disclosure.

[0061] Referring to FIG. 3, the electronic device (100) includes a camera (110), a driving unit (120), a memory (130), and at least one processor (140).

[0062] According to one embodiment, the camera (110) is configured to capture a subject and obtain a captured image, wherein the image may include a static image or a dynamic image.

[0063] In particular, the camera (110) can capture various gestures of a subject located within the space. The camera (110) may be provided in at least one of the top area, bottom area, and side area of ​​the electronic device (100) to capture the subject. The camera (110) may be implemented as a single camera, and may also be implemented as a plurality of cameras according to the embodiment.

[0064] Additionally, the camera (110) can provide the captured image of the subject to the processor (140) to identify the subject gesture within the captured image.

[0065] Additionally, the camera (110) may be implemented as a wide-angle camera to capture a wide field of view, but is not limited thereto.

[0066] According to one embodiment, the driving unit (120) is configured to move the electronic device (100). To this end, the driving unit (120) includes a motor, a wheel, etc., and can move the electronic device (100) through the movement of the wheel.

[0067] Meanwhile, the driving unit (120) can adjust the shooting direction of the electronic device (100). For example, the direction in which the camera (110) looks can be adjusted by adjusting the body position of the electronic device (100) as shown in FIG. 2, or the shooting direction can be adjusted by adjusting the lens position within the camera (110).

[0068] Here, the shooting direction refers to the direction in which the subject is photographed, and may also be referred to as the direction the electronic device faces, the projection direction, etc.

[0069] According to an embodiment, the memory (130) may store various programs, data, instructions, etc. used in the electronic device (100). In addition, the memory (130) may store various information according to various embodiments of the present disclosure.

[0070] The memory (130) may be implemented in the form of a memory embedded in the electronic device (100) or in the form of a memory that can be attached to and detached from the electronic device (100), depending on the purpose of data storage.

[0071] For example, data for driving the electronic device (100) may be stored in memory embedded in the electronic device (100), and data for the expansion function of the electronic device (100) may be stored in memory that is detachable from the electronic device (100).

[0072] In the case of memory embedded in an electronic device (100), it may be implemented in the form of volatile memory (e.g., DRAM (dynamic RAM), SRAM (static RAM), or SDRAM (synchronous dynamic RAM), etc.), non-volatile memory (e.g., OTPROM (one time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), etc.), hard drive, or solid state drive (SSD).

[0073] In the case of a memory that can be attached to and detached from an electronic device (100), it can be implemented in the form of a memory card (e.g., CF (compact flash), SD (secure digital), Micro-SD (micro secure digital), Mini-SD (mini secure digital), xD (extreme digital), MMC (multi-media card), etc.) or an external memory that can be connected to a USB port (e.g., USB memory).

[0074] The memory (130) may include various instructions required for the operation of the processor (140). Here, the instructions may include instructions for acquiring an image corresponding to a subject, instructions for identifying that driving of an electronic device is required based on subject information included in the image, instructions for keystone correction, instructions for controlling a driving unit to drive an electronic device, instructions for processing an image, etc.

[0075] The memory (130) can store registered gestures. Here, a registered gesture may refer to a gesture that is previously registered in an electronic device (100) or server to perform an action corresponding to a registered gesture when a subject gesture recognized through the camera (110) is identified as a registered gesture stored in the memory (130).

[0076] However, it is not limited to this, and the registered gesture may refer to a gesture for adjusting the direction or position of the electronic device (100) in order to reduce the occurrence of errors by pre-registering gestures that have a high frequency of errors in gesture identification and identifying gestures with a high probability of error.

[0077] Subject gestures can refer to body movements using the subject's hands, arms, face, etc., that convey a specific meaning. For example, subject gestures may include actions such as waving a hand or giving a thumbs up.

[0078] Meanwhile, the subject refers to an object or person that is the target of being photographed through a camera, and in the present disclosure, the subject may refer to a person making a gesture.

[0079] Here, the subject may be referred to as the user, and the subject gesture may be referred to as the user gesture or input gesture.

[0080] According to an embodiment, at least one processor (140) controls the overall operation of the electronic device (100). Specifically, at least one processor (140) is connected to each component of the electronic device (100) and can control the overall operation of the electronic device (100).

[0081] At least one processor (140) can perform the operation of an electronic device (100) according to various embodiments by executing at least one instruction stored in memory.

[0082] According to an embodiment, at least one processor (140) may be implemented as a digital signal processor (DSP) that processes digital signals, a microprocessor, or a TCON (Timing controller). However, it is not limited thereto and may include or be defined by one or more of a central processing unit (CPU), a Micro Controller Unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a communication processor (CP), an ARM processor, or an AI (Artificial Intelligence) processor. Additionally, at least one processor (140) may be implemented as a System on Chip (SoC) or Large Scale Integration (LSI) with a built-in processing algorithm, or may be implemented in the form of a Field Programmable Gate Array (FPGA). At least one processor (140) can perform various functions by executing computer executable instructions stored in memory.

[0083] At least one processor (140) may include one or more of a CPU (Central Processing Unit), GPU (Graphics Processing Unit), APU (Accelerated Processing Unit), MIC (Many Integrated Core), DSP (Digital Signal Processor), NPU (Neural Processing Unit), hardware accelerator, or machine learning accelerator.

[0084] At least one processor (140) can acquire an image corresponding to a subject through a camera (110) and identify whether driving of an electronic device is required based on the subject information included in the image.

[0085] For example, at least one processor (140) can identify a situation where driving of the electronic device is required if the direction in which the subject in the captured image takes a gesture is a direction in which the gesture is difficult to identify.

[0086] Here, the subject information may include at least one of the subject's facial orientation, a pose corresponding to the subject, and a subject gesture.

[0087] At least one processor (140) can control the drive unit (120) to move the electronic device if movement is required based on the subject information.

[0088] For example, at least one processor (140) can control the driving unit (120) to change the scanning direction of the camera (110) based on subject information. Here, the scanning direction of the camera may mean the direction in which the camera sensor scans the image.

[0089] Specifically, at least one processor (140) can control the driving unit (120) so that if the direction of the subject's face is not aligned with the scanning direction of the camera (110), the direction of the subject's face and the scanning direction of the camera (110) are aligned with each other.

[0090] At least one processor (140) can control a driving unit (120) to change the position of an electronic device (100) based on subject information and to change the scanning direction of a camera (110) based on the changed position information of the electronic device (100) and subject information. This is explained in detail in FIG. 5.

[0091] At least one processor (140) can control one or any combination of other components of the electronic device and can perform operations or data processing related to communication. At least one processor (140) can execute one or more programs or instructions stored in memory.

[0092] For example, at least one processor (140) can perform the method according to an embodiment of the present disclosure by executing one or more instructions stored in memory.

[0093] If the method according to the embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by a single processor or by a plurality of processors.

[0094] For example, when the first operation, the second operation, and the third operation are performed by the method according to the embodiment, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence dedicated processor).

[0095] One or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory (130). Alternatively, if one or more processors are dedicated artificial intelligence processors, the dedicated artificial intelligence processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model. The predefined operation rules or artificial intelligence models are characterized by being created through learning.

[0096] Here, being created through learning means that a basic artificial intelligence model is trained using multiple training data by a learning algorithm, thereby creating predefined behavioral rules or an artificial intelligence model configured to perform a desired characteristic (or objective).

[0097] Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0098] An artificial intelligence model can be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through calculations between the results of the previous layer and the multiple weights.

[0099] Multiple weights possessed by multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, multiple weights can be updated so that the loss or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized.

[0100] Artificial neural networks may include deep neural networks (DNNs), such as, but are not limited to, Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), Generative Adversarial Networks (GANs), or Deep Q-Networks.

[0101] At least one processor (140) may be implemented as a single core processor including one core, or as one or more multicore processors including multiple cores (e.g., homogeneous multicore or heterogeneous multicore).

[0102] When at least one processor (140) is implemented as a multi-core processor, each of the multiple cores included in the multi-core processor may include internal processor memory such as cache memory or on-chip memory, and a common cache shared by multiple cores may be included in the multi-core processor.

[0103] Each of the multiple cores (or some of the multiple cores) included in the multi-core processor may independently read and execute program instructions for implementing the method according to the embodiment of the present disclosure, or all (or some of) of the multiple cores may be linked together to read and execute program instructions for implementing the method according to the embodiment of the present disclosure.

[0104] FIG. 4 is a detailed block diagram for illustrating an electronic device according to at least one embodiment of the present disclosure.

[0105] Referring to FIG. 4, an electronic device (100) according to one embodiment of the present disclosure may include a camera (110), a driving unit (120), a memory (130), at least one processor (140), a projection device (150), and an input interface (160). Parts that overlap with the description above will be omitted or abbreviated below.

[0106] The projection device (150) is configured to project light containing an image onto an external screen or wall to display an image. The projection device (150) can change the brightness of the image by adjusting the intensity of the light containing the image.

[0107] Specifically, the projection device (150) can project an image or video, including at least one of an image received from a source device and an image stored in advance, onto a projection area using a light source such as a lamp or an LED.

[0108] The projection device (150) can project an image onto the front of the electronic device (100).

[0109] In this case, the projection device (150) may change and project the image based on keystone correction information obtained from at least one processor (140) in order to maintain the shape of the projected image.

[0110] Correction according to one embodiment of the present disclosure may include an operation in which an electronic device improves, modifies, or enhances the image quality, characteristics, etc.

[0111] For example, at least one processor (140) may perform corrections to enhance the texture, edges, etc. of an image to improve the characteristics of the image. For example, at least one processor (140) may perform corrections to upscale the image to improve the image quality, resolution, etc.

[0112] Meanwhile, at least one processor (140) can perform various corrections in addition to the operation of correcting the image.

[0113] For example, if the electronic device is implemented as a modular display device including a plurality of modules, at least one processor (140) can obtain a correction coefficient for correcting the brightness, color, etc. of each of the plurality of modules to resolve non-uniformity between the plurality of modules and to make the image output by the modular display device uniform.

[0114] According to one example, at least one processor (140) can output a uniform image by correcting the brightness, color, brightness, etc. of each of the plurality of modules based on a correction coefficient corresponding to each of the plurality of modules.

[0115] For example, at least one processor (140) may correct the position, direction, etc. of the image output. For example, if the electronic device is implemented as a projector, at least one processor (140) may correct the position, direction, etc. of the image output by performing lens shift, color matching, edge blending, keystone correction, leveling correction (horizontal correction), focus correction, etc.

[0116] In the present disclosure, keystone correction may mean correction to remove distortion when an image projected through a projector device or an image acquired from a camera is distorted.

[0117] The input interface (160) can receive various feedback from the user. For example, if at least one processor (140) receives user input regarding a setting to control each configuration of the electronic device (100) through the input interface (160), it can operate each configuration in response to the user input.

[0118] For example, when at least one processor (140) provides a plurality of candidate gesture information among registered gestures stored in memory (130) to a user, the input interface (160) can receive user input selecting some of the plurality of candidate gestures. This is explained in detail in FIG. 8.

[0119] The input interface (160) may include a microphone, a touchscreen, etc., but is not limited thereto, and may include various input interfaces capable of receiving user input.

[0120] Here, the microphone is a component for receiving sound input and converting it into an audio signal. The microphone is electrically connected to at least one processor (140) and can receive sound under the control of at least one processor (140).

[0121] For example, the microphone may be formed as an integrated unit on the upper side, front side, or side side of the electronic device (100). Alternatively, the microphone may be provided in a remote control or the like, separate from the electronic device (100). In this case, the remote control may receive sound through the microphone and provide the received sound to the electronic device (100).

[0122] A microphone may include various components such as a microphone that collects analog sound, an amplifier circuit that amplifies the collected sound, an A / D converter circuit that samples the amplified sound and converts it into a digital signal, and a filter circuit that removes noise components from the converted digital signal.

[0123] Meanwhile, the microphone may be implemented in the form of a sound sensor, and any configuration capable of collecting sound is acceptable.

[0124] A communication interface (not shown) can communicate with an external server or an external electronic device. In particular, the communication interface can receive information about a registration gesture from an external server or an external electronic device.

[0125] The communication interface may include wired or wireless input / output interfaces (or input / output terminals) according to various standards. For example, one or more connection interfaces may include various interfaces such as HDMI (High Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB (D-subminiature), DVI (Digital Visual Interface), AP-based Wi-Fi (Wi-Fi, Wireless LAN Network), Bluetooth, Zigbee, wired / wireless LAN (Local Area Network), WAN (Wide Area Network), Ethernet, IEEE 1394, AES / EBU (Audio Engineering Society / European Broadcasting Union), Optical, Coaxial, etc.

[0126] Meanwhile, regarding the communication interface, the communication interface communicating with the server and the communication interface communicating with the remote control may be different or the same.

[0127] For example, at least one processor (140) may transmit a video or audio signal to an external display device through a communication interface. In this case, an image may be output through the external device. The electronic device (100) or the external device may each have a port for a separate output.

[0128] The operation of the electronic device (100) is described in more detail below through FIGS. 5 to 9. FIGS. 5 to 9 describes individual embodiments for convenience of explanation. However, the individual embodiments of FIGS. 5 to 9 may be implemented in any combination.

[0129] FIG. 5 is a drawing for explaining the operation of an electronic device running according to at least one embodiment of the present disclosure.

[0130] Referring to FIG. 5, at least one processor (140) can control the driving unit (120) to change the scanning direction or change the position of the electronic device (100) if the facial direction of the subject and the scanning direction of the camera do not face each other in a straight line.

[0131] For example, at least one processor (140) can change the scanning direction of the camera (110) by rotating the body of the electronic device (100) to the left so that it faces in a straight line when the subject (10) is not located in a straight line with the electronic device (100) but is located to the left.

[0132] At least one processor (140) can identify that driving is required if, even after changing the scanning direction of the camera (110), the face direction of the subject in the captured image does not face in a straight line with the scanning direction of the camera (110).

[0133] At least one processor (140) can control a driving unit (120) to change the position of an electronic device (100) and change the scanning direction of a camera (110) based on the position information of the changed electronic device (100) and subject information.

[0134] At least one processor (140) can obtain information about a minimized movement path of the electronic device (100) based on subject information including the facial direction of the subject (10) when a change in the position of the electronic device (100) is required. At least one processor (140) can control the driving unit (120) to operate based on the subject information and the minimized movement path.

[0135] Referring to FIG. 5, at least one processor (140) can control a projection device (150) to project an image onto the front of an electronic device (100) while driving in space based on subject information. At least one processor (140) can perform keystone correction to maintain the shape of the projected image while driving. This is explained in detail in FIG. 6.

[0136] Meanwhile, although the present disclosure describes the scanning direction of the camera (110) as a reference, it is not limited thereto, and at least one processor (140) may identify whether driving is required based on the projection direction of the projection device (150).

[0137] FIG. 6 is a flowchart illustrating the overall operation of an electronic device according to at least one embodiment of the present disclosure.

[0138] Referring to FIG. 6, when a subject performs a gesture input (610), the electronic device (100) can obtain a captured image by photographing the subject through a camera (110) and obtain subject information through image analysis (620).

[0139] Here, acquisition may include an operation to obtain specific data. In this case, acquisition may include an operation to acquire data by receiving it, an operation to acquire data by identifying it, an operation to acquire data by deriving it, an operation to acquire data by determining it, an operation to acquire data by analyzing it, an operation to acquire data by detecting it, an operation to acquire data by searching for it, an operation to acquire data by recognizing it, an operation to acquire data by detecting it, an operation to acquire data by distinguishing it, an operation to acquire data by classifying it, and an operation to acquire data by separating it.

[0140] At least one processor (140) can identify (630) a situation requiring driving if the face direction of the subject in the captured image is not aligned with the scanning direction of the camera (110).

[0141] If at least one processor (140) identifies a situation where driving is required, driving (640) can be performed in two modes.

[0142] For example, the electronic device (100) can drive by selecting at least one mode among a first mode that rotates based on the angle of the camera (110) or a second mode that moves based on the distance between the electronic device (100) and the subject.

[0143] The electronic device (100) obtains information about a minimized movement path based on subject information and can identify whether additional driving is required after driving in a first mode based on the subject information and the movement path. If the electronic device (100) identifies that additional driving is required after driving in the first mode, it can drive in a second mode.

[0144] However, it is not limited to this, and the electronic device (100) may drive in the first mode depending on whether additional driving is required after driving in the second mode.

[0145] Meanwhile, if at least one processor (140) identifies a situation where driving is required, it may perform keystone correction (640) in real time while driving in two modes.

[0146] For example, at least one processor (140) controls a projection device (150) to project an image onto the front of an electronic device (100), and while the electronic device (100) is moving, it can obtain keystone correction information for the image to maintain the shape of the projected image.

[0147] At least one processor (140) can control the projection device (150) to change and project the image based on keystone correction information in order to maintain the shape of the image projected while driving.

[0148] FIG. 7 is a flowchart illustrating the overall operation of an electronic device according to at least one embodiment of the present disclosure.

[0149] Referring to FIG. 7, when a subject performs a gesture input (710), the electronic device (100) can obtain a captured image by photographing the subject through a camera (110) and obtain subject information through image analysis (720).

[0150] At least one processor (140) can determine whether the input gesture and the registration gesture are similar (730) and identify whether the input gesture is the registration gesture.

[0151] Here, whether it is similar to a registered gesture can be identified as similar if it exceeds a pre-set similarity level, but is not limited thereto.

[0152] For example, at least one processor (140) can identify whether a subject gesture is a registered gesture based on the similarity between a subject gesture recognized through a camera (110) and a registered gesture stored in memory (130).

[0153] However, it is not limited to this, and various image analysis techniques may also be applied.

[0154] At least one processor (140) can identify (740) a situation requiring driving if the subject gesture is not identified as a registered gesture.

[0155] If at least one processor (140) identifies a situation where driving is required, driving (750) can be performed in two modes.

[0156] For example, the electronic device (100) can drive by selecting at least one mode among a first mode that rotates based on the angle of the camera (110) or a second mode that moves based on the distance between the electronic device (100) and the subject.

[0157] The electronic device (100) obtains information about a minimized movement path based on subject information and can identify whether additional driving is required after driving in a first mode based on the subject information and the movement path. If the electronic device (100) identifies that additional driving is required after driving in the first mode, it can drive in a second mode.

[0158] However, it is not limited to this, and the electronic device (100) may drive in the first mode depending on whether additional driving is required after driving in the second mode.

[0159] Meanwhile, if at least one processor (140) identifies a situation where driving is required, it may perform keystone correction (750) in real time while driving in two modes.

[0160] For example, at least one processor (140) controls a projection device (150) to project an image onto the front of an electronic device (100), and while the electronic device (100) is moving, it can obtain keystone correction information for the image to maintain the shape of the projected image.

[0161] At least one processor (140) can control the projection device (150) to change and project the image based on keystone correction information in order to maintain the shape of the image projected while driving.

[0162] Meanwhile, at least one processor (140) can perform an action (760) corresponding to the registered gesture if it is identified that the input gesture and the registered gesture are similar.

[0163] For example, if a subject gesture recognized through the camera (110) is identified as a registered gesture stored in the memory (130), an action corresponding to the registered gesture (e.g., moving forward, backward, rotating, turning image projection on and off, etc.) can be performed.

[0164] FIG. 8 is a diagram illustrating the operation of an electronic device providing gesture information according to at least one embodiment of the present disclosure.

[0165] Referring to FIG. 8, if the input user gesture is not identified as a registered gesture, the electronic device (100) may perform an action corresponding to a selected gesture by receiving input from the user through providing information on a plurality of candidate gestures among the registered gestures to the user.

[0166] For example, if a subject gesture recognized through the camera (110) is not identified as a registered gesture, at least one processor (140) may provide information on a plurality of candidate gestures among the registered gestures stored in memory (130) and receive user input selecting some of the plurality of candidate gestures through an input interface (160).

[0167] For example, referring to FIG. 8, when a user speaks "execute the left one", the electronic device (100) can receive user input selecting the left gesture (82-1) among candidate gestures projected as an image through a microphone.

[0168] However, it is not limited to this, and user input may also be received through various input interfaces such as touch screens and camera sensors.

[0169] Additionally, user input may be received through an external device. For example, the electronic device (100) may control each component of the electronic device (100) or perform voice recognition functions using an app installed on an external device such as a smartphone.

[0170] At least one processor (140) can perform an action corresponding to a registered gesture based on user input when it receives user input.

[0171] The electronic device (100) may obtain multiple candidate gesture information using data for registered gestures stored in memory (130) as well as a database for registered gestures stored on an external server.

[0172] FIG. 9a is a drawing for explaining the operation of an electronic device providing guide information according to at least one embodiment of the present disclosure.

[0173] The electronic device (100) can analyze the user's location within the captured image and provide the user with a guide to change the user's location.

[0174] For example, if the electronic device (100) does not identify the subject gesture recognized through the camera (110) as a registered gesture, it may provide guide information to change the position of the subject based on the subject information.

[0175] For example, referring to FIG. 9a, when the facial direction of the subject (10) and the scanning direction of the camera (110) of the electronic device (100) do not face each other in a straight line, the electronic device (100) can output a guide voice such as “Please move to the right” (91) through a speaker.

[0176] However, it is not limited to this, and the electronic device (100) may provide guide information through various output interfaces such as a display, a projection device, in addition to a speaker. This is explained in detail in FIG. 9b.

[0177] Meanwhile, the electronic device (100) not only guides the direction of movement of the subject but can also guide the degree of movement required or the change in facial angle by expressing it numerically.

[0178] For example, if the electronic device (100) moves to the right according to the guide “Please move to the right” (91), it may output an additional guide voice “Please rotate 10 degrees clockwise” (92) so that the direction of the subject’s face corresponds to the direction of the camera.

[0179] FIG. 9b is a drawing for explaining the operation of an electronic device providing guide information according to at least one embodiment of the present disclosure.

[0180] The electronic device (100) may provide guide information to the user by displaying different LED lengths through an LEC output device installed in the electronic device (100) based on the position information of the subject.

[0181] For example, in FIG. 9a, if the facial direction of the subject (10) and the scanning direction of the camera (110) of the electronic device (100) do not face each other in a straight line, the electronic device (100) can indicate that the length of the right LED (90-1) of the two LEDs is longer as a guide to indicate that movement to the right is required.

[0182] Meanwhile, as shown in FIG. 9a, the electronic device (100) may change the length of the right LED (90-2) of the two LEDs to be equal to the length of the left LED when the subject moves to the right according to the guide of the electronic device (100).

[0183] FIG. 10 is a flowchart illustrating the operation of an electronic device running according to at least one embodiment of the present disclosure.

[0184] Referring to FIG. 10, the electronic device can acquire an image corresponding to a subject (S1010). For example, when the subject makes a specific gesture, the electronic device can acquire an image corresponding to the subject through a camera.

[0185] The electronic device can obtain subject information included in the image (S1020). For example, through image analysis, subject information such as the facial orientation of the subject in the captured image, the pose corresponding to the subject, and the subject gesture can be obtained.

[0186] If movement is required based on subject information, the electronic device can drive for movement (S1030). For example, if the electronic device identifies a situation where the subject's gesture is difficult to recognize based on the subject's facial direction and the camera's scanning direction, it may identify that driving is required.

[0187] However, this is not limited thereto, and the electronic device may identify that driving is required even when the input subject gesture is not identified as a registered gesture based on the similarity between the subject gesture and the registered gesture.

[0188] The driving method of the electronic device described in FIG. 10 can be performed by a device having various configurations such as FIG. 3 and FIG. 4 described above, but is not necessarily limited thereto and can be performed by a device having various configurations.

[0189] The various embodiments described above may be implemented as individual embodiments, or at least one embodiment may be combined with one another, either wholly or partially, to be implemented together in a single device.

[0190] According to the various embodiments described above, a movable electronic device can perform actions corresponding to the user's intention by moving to an optimal position to accurately recognize the user's gesture. Ultimately, this can enhance the user's experience.

[0191] Various embodiments of the present disclosure may be implemented as software stored on a machine-readable storage media that can be mounted on or connected to a smartphone, a user terminal device, and various other electronic devices (e.g., a computer).

[0192] Specifically, a non-transient readable storage medium may be provided that stores software for sequentially performing the steps of acquiring an image corresponding to a subject, acquiring subject information included in the image, and, if movement is required based on the subject information, moving an electronic device.

[0193] A device equipped with such a non-transient readable medium can perform various operations such as tag identification corresponding to user actions described in the various embodiments above, verification of importance per multiple frames, and generation of edited images.

[0194] In non-transitory readable storage media, 'non-transitory' simply means that the storage medium does not contain a signal and is tangible; it does not distinguish whether data is stored semi-permanently or temporarily on the storage medium.

[0195] Alternatively, a program for performing the methods according to the various embodiments described above may be distributed online through an application store. In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a storage medium such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0196] Each component (e.g., module or program) according to various embodiments may consist of a single or multiple entities, and some of the aforementioned sub-components may be omitted, or other sub-components may be further included in various embodiments. Generally or additionally, some components (e.g., module or program) may be integrated into a single entity to perform the same or similar functions as those performed by each of the respective components prior to integration.

[0197] Operations performed by a module, program, or other component according to various embodiments may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0198] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

1. In an electronic device, camera; A driving unit for moving the above electronic device; Memory for storing at least one instruction; and It includes at least one processor that executes the above at least one instruction; and The above-mentioned at least one processor is, An image corresponding to the subject is obtained through the above camera, and If movement is required based on the subject information included in the above image, the drive unit is controlled for movement, and The above subject information is, An electronic device comprising at least one of the facial direction of the subject, a pose corresponding to the subject, and a gesture of the subject.

2. In Paragraph 1, The above-mentioned at least one processor is, An electronic device that controls the driving unit to change the camera direction based on the above subject information.

3. In Paragraph 2, The above-mentioned at least one processor is, An electronic device that controls the driving unit so that the facial direction of the subject and the camera direction correspond when the facial direction of the subject does not correspond to the camera direction.

4. In Paragraph 1, The above-mentioned at least one processor is, The driving unit is controlled to change the position of the electronic device based on the above subject information, and An electronic device that controls the driving unit to change the camera direction based on position information corresponding to the changed position and the subject information.

5. In Paragraph 1, It further includes a projection device; and The above-mentioned at least one processor is, Control the projection device to project an image onto an area corresponding to the front of the electronic device, and Keystone correction information corresponding to maintaining the shape of an image projected through the projection device while moving by controlling the above driving unit is obtained, and An electronic device that controls the projection device to project a modified image based on the above keystone correction information.

6. In Paragraph 1, The above-mentioned at least one processor is, An electronic device that controls the performance of an action corresponding to a registered gesture when the subject gesture included in the subject information corresponds to a registered gesture stored in the memory.

7. In Paragraph 6, The above-mentioned at least one processor is, An electronic device that identifies whether a subject gesture corresponds to a registered gesture based on the similarity between the subject gesture included in the subject information and the registered gesture stored in the memory.

8. In Paragraph 1, It further includes an input interface, The above-mentioned at least one processor is, If the subject gesture included in the subject information does not correspond to a registered gesture stored in the memory, control is provided to provide a plurality of candidate gesture information among the registered gestures. An electronic device that, upon receiving user input corresponding to the selection of one of the plurality of candidate gestures through the input interface, performs an action corresponding to the registered gesture based on the user input.

9. In Paragraph 1, The above-mentioned at least one processor is, An electronic device that controls to provide guide information corresponding to a change in the position of the subject based on the subject information if the subject gesture included in the subject information does not correspond to a registered gesture stored in the memory.

10. In Paragraph 1, The above-mentioned at least one processor is, Based on the subject information including the facial direction of the subject, information corresponding to the minimum movement path is obtained, and An electronic device that controls the driving unit to move based on the above subject information and information corresponding to the above movement path.

11. In a method for controlling an electronic device, A step of acquiring an image corresponding to the subject; A step of acquiring subject information within the above image; and If movement is required based on the above subject information, the electronic device moves; including the step of moving. The above subject information is, A control method comprising at least one of the facial direction of the subject, a pose corresponding to the subject, and a subject gesture.

12. In Paragraph 11, The step of moving the above electronic device is, A control method comprising the step of changing the camera direction of the electronic device based on the subject information.

13. In Paragraph 12, The step of moving the above electronic device is, A control method comprising: a step of changing the camera direction so that the face direction of the subject and the camera direction correspond if the face direction of the subject does not correspond to the camera direction.

14. In Paragraph 11, The step of moving the above electronic device is, A step of changing the position of the electronic device based on the above subject information; and A control method comprising the step of changing the camera direction of the electronic device based on position information corresponding to the changed position and the subject information.

15. A non-transient readable recording medium comprising a program for executing a method of controlling an electronic device, The control method of the above electronic device is, A step of acquiring an image corresponding to the subject; A step of acquiring subject information within the above image; and If movement is required based on the above subject information, the electronic device moves; including the step of moving. The above subject information is, A non-transient readable recording medium comprising at least one of the facial direction of the subject, a pose corresponding to the subject, and a gesture of the subject.

Images