Electronic device and control method therefor

Abstract

Disclosed is an electronic device. The electronic device comprises: a camera; a driving device; a memory for storing at least one instruction; and at least one processor for executing the at least one instruction. The at least one processor: acquires body information corresponding to a subject on the basis of a first image acquired by means of the camera; controls the driving device to move to a position corresponding to the face direction of the subject on the basis of the body information; acquires facial data corresponding to the subject on the basis of a second image acquired by means of the camera; and acquires identification information corresponding to the subject on the basis of the body information and the facial data.

Description

Electronic device and control method thereof

[0001] The present disclosure relates to an electronic device for acquiring identification information of a subject and a method for controlling the same.

[0002] Thanks to advancements in electronic technology, various types of electronic devices are being used in daily life. Among these devices may be electronic devices that travel through space using a driving mechanism.

[0003] For example, there may be a mobile electronic device that travels through space and acquires identification information about a subject when the subject is identified.

[0004] Identification information refers to information that can uniquely identify a specific individual or object.

[0005] According to at least one embodiment of the present disclosure, an electronic device comprises a camera, a driving device, a memory for storing at least one instruction, and at least one processor for executing at least one instruction. The at least one processor may acquire body information corresponding to a subject based on a first image acquired through the camera, control the driving device to move to a position corresponding to the direction of the subject's face based on the body information, acquire face data corresponding to the subject based on a second image acquired through the camera, and acquire identification information corresponding to the subject based on the body information and face data.

[0006] Additionally, according to at least one embodiment of the present disclosure, the control method may include the steps of: acquiring body information corresponding to a subject based on a first image acquired through the electronic device; moving to a position corresponding to the direction of the subject's face based on the body information; acquiring face data corresponding to the subject based on a second image acquired through the electronic device; and acquiring identification information of the subject based on the body information and the face data.

[0007] Additionally, a non-transient readable recording medium according to at least one embodiment of the present disclosure stores a program for performing a control method of an electronic device, comprising the steps of: acquiring body information corresponding to a subject based on a first image acquired through the electronic device; moving to a position corresponding to the direction of the face of the subject based on the body information; acquiring face data corresponding to the subject based on a second image acquired through the electronic device; and acquiring identification information of the subject based on the body information and the face data.

[0008] FIG. 1 is a drawing for explaining the operation of an electronic device according to at least one embodiment of the present disclosure.

[0009] FIG. 2 is a drawing illustrating the form of an electronic device according to at least one embodiment of the present disclosure.

[0010] FIG. 3 is a block diagram illustrating the configuration of an electronic device according to at least one embodiment of the present disclosure.

[0011] FIG. 4 is a detailed block diagram for illustrating an electronic device according to at least one embodiment of the present disclosure.

[0012] FIG. 5 is a drawing for explaining the operation of a processor of an electronic device according to at least one embodiment of the present disclosure.

[0013] FIG. 6 is a drawing for explaining the operation of an electronic device according to at least one embodiment of the present disclosure to identify and move a subject.

[0014] FIG. 7 is a flowchart illustrating the overall operation of an electronic device according to at least one embodiment of the present disclosure.

[0015] FIG. 8 is a flowchart illustrating the operation of an electronic device according to at least one embodiment of the present disclosure to additionally learn facial data.

[0016] FIG. 9 is a flowchart illustrating the operation of an electronic device according to at least one embodiment of the present disclosure to additionally learn body data.

[0017] FIG. 10 is a flowchart illustrating the operation of an electronic device according to at least one embodiment of the present disclosure identifying a subject based on the subject's body information.

[0018] FIG. 11 is a table for explaining the operation of an electronic device according to at least one embodiment of the present disclosure applying weights to data.

[0019] FIG. 12 is a flowchart illustrating the operation of an electronic device acquiring identification information according to at least one embodiment of the present disclosure.

[0020] The terms used in the various embodiments of this Disclosure have been selected to be as widely used and general as possible, taking into account their functions within this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been selected at the applicant's discretion, and in such cases, their meanings will be described in detail in the relevant description section of this disclosure. Therefore, terms used in this disclosure should be defined not merely by their names, but based on their meanings and the overall content of this disclosure.

[0021] The various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features described in the present disclosure to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments.

[0022] In relation to the description of the drawings, similar reference numerals may be used for similar or related components.

[0023] The singular form of the noun corresponding to the item may include one or multiple items, unless the relevant context clearly indicates otherwise.

[0024] In the present disclosure, each of the phrases such as “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “at least one of A, B, or C” may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

[0025] Terms such as "first," "second," or "first" or "second" may be used simply to distinguish a component from another component and do not limit the components in other aspects (e.g., importance or order).

[0026] Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that the component may be connected to the other component directly (e.g., via a wire), wirelessly, or through a third component.

[0027] Terms such as "include" or "have" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in this disclosure, and do not preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0028] When it is said that a component is "connected," "combined," "supported," or "in contact" with another component, this includes not only cases where the components are directly connected, combined, supported, or in contact, but also cases where they are indirectly connected, combined, supported, or in contact through a third component.

[0029] When it is said that a component is located "on" another component, this includes not only cases where one component is in contact with the other, but also cases where another component exists between the two components.

[0030]

[0031] *20 The term "and / or" includes a combination of multiple related described components or any of the multiple related described components.

[0032] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Additionally, a plurality of "modules" or a plurality of "parts" may be integrated into at least one module and implemented by at least one processor, except for a "module" or "part" that needs to be implemented in specific hardware.

[0033] Meanwhile, various elements and areas in the drawings are depicted schematically. Accordingly, the technical concept of the present disclosure is not limited by the relative sizes or spacing depicted in the attached drawings.

[0034] In the present disclosure, the term "user" may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0035] An embodiment of the present disclosure will be described in more detail below with reference to the attached drawings.

[0036] FIG. 1 is a drawing for explaining the operation of an electronic device according to at least one embodiment of the present disclosure.

[0037] Referring to FIG. 1, the electronic device (100) can identify a subject (10) while traveling through space. Here, "space" may refer to a space where the electronic device (100) can travel.

[0038] For example, when a subject (10) is identified, the electronic device (100) may acquire a first image of the subject's body and acquire body information corresponding to the subject based on the first image. Based on the acquired body information, the electronic device (100) may move to a position corresponding to the direction of the subject's face.

[0039] Here, "driving" may include the action of an object moving using power.

[0040] According to embodiments of the present disclosure, driving may include the movement of an object moving in any direction through power. Alternatively, driving may include the movement of an object moving along a preset track through power.

[0041] For example, driving may include the action of changing the position of an electronic device during the process of acquiring a captured image of a subject.

[0042] Here, "body information" may refer to information about the subject's body, including static or dynamic data. "Static data" may correspond to data related to the subject's body. "Dynamic data" may correspond to data regarding the subject's gait.

[0043] Here, when the electronic device (100) moves to a position corresponding to the direction of the subject's face based on body information, it may include a case where the similarity between the body information and the registered data exceeds a preset value.

[0044] For example, if the camera direction of the electronic device (100) deviates from a straight line with the direction of the subject's face (10), an error in misidentifying the subject (10) may occur, so the electronic device (100) can move to a position corresponding to the direction of the subject's face in order to accurately identify the subject (10).

[0045] Meanwhile, when multiple objects (11, 12, 13) are identified during driving, the electronic device (100) can acquire a captured image of the multiple objects (11, 12, 13) and determine whether to drive by comparing the similarity between the object data in the captured image and the registered data stored in the database.

[0046] For example, the electronic device (100) can compare the body information in the captured image of a specific object (11) among a plurality of objects (11, 12, 13) with the registered data stored in the database, and if it is identified that it has a similarity exceeding a preset value, it can move to a position corresponding to the face direction of the specific object (11).

[0047] In FIG. 1, the electronic device (100) is shown in the form of a portable projector, but is not limited thereto, and the electronic device (100) can be implemented as various types of electronic devices such as a portable electronic device or an electronic device that can be carried by a user.

[0048] Here, a projector refers to a device that displays an image by projecting light containing an image onto an external screen or wall, and a portable projector refers to a projector that can be moved using wheels, motors, etc.

[0049] Here, the subject refers to an object or animal, including a person, that is the target of being photographed through a camera, and in the present disclosure, the subject may refer to a person, and the subject may also be referred to as a user.

[0050] The subject depicted in Fig. 1 is shown in the form of a human, but is not limited thereto, and the subject may include animals such as dogs and cats.

[0051] Hereinafter, the operation of an electronic device (100) according to various embodiments of the present disclosure will be described.

[0052] FIG. 2 is a drawing illustrating the form of an electronic device according to at least one embodiment of the present disclosure.

[0053] Referring to FIG. 2, a movable electronic device (100) is illustrated. This is merely an example of a disclosed form and may have a shape different from the illustrated form.

[0054] The electronic device (100) can obtain a captured image of a subject through a camera (110).

[0055] An image according to one example may include a captured image acquired using a sensor (e.g., a camera) equipped in an electronic device, an input image received from an external device through a communication unit, a graphic image generated by the electronic device, etc.

[0056] Images according to one example may include, depending on the aspect ratio, horizontal images where the width is longer than the height (e.g., landscape images, horizontal images), vertical images where the height is longer than the width (e.g., portrait images, vertical images), etc. For example, horizontal images may include images with a 16:9 aspect ratio, and vertical images may include images with a 9:16 aspect ratio. Specific numbers are examples for convenience of explanation and are not limited thereto.

[0057] An image according to one example may include various resolutions depending on the number of pixels constituting the image (the product of the number of pixels in the horizontal direction and the number of pixels in the vertical direction). For example, depending on the resolution, the image may include high-resolution images (FHD (1920Y1080), 8K (7680Y4320), etc.) and low-resolution images (640Y480, etc.).

[0058] The electronic device (100) can travel through space based on subject information within a captured image through a driving device (120).

[0059] For example, the electronic device (100) can move to a position corresponding to the direction of the subject's face to obtain identification information of the subject.

[0060] Here, "identification information of the subject" may refer to unique personal information capable of identifying the subject. For example, the identification information of the subject may include the subject's physical characteristics, the subject's personal information (e.g., name, date of birth, etc.).

[0061] A shooting according to one example of the present disclosure may include the operation of an electronic device that controls a camera (e.g., a camera including an image sensor and a lens) equipped in an electronic device to convert an optical image formed through a lens into an electrical signal and acquire an image.

[0062] For example, an electronic device (100) can control a camera (110) equipped in the electronic device to photograph the surroundings of the electronic device and obtain an image (e.g., a captured image) containing one or more frames. Here, the image may include a live-view image.

[0063] The electronic device (100) includes a driving device (120) for movement (e.g., a motor, a wheel) and can perform movement using the provided driving device. Meanwhile, although the illustrated example is illustrated and described as moving the electronic device (100) using a wheel, it is also possible to use other means in the form of a caterpillar when implementing it, and if the electronic device (100) is implemented as a drone, it is also possible to equip a propeller instead of a wheel.

[0064] Meanwhile, in the illustrated example, the electronic device (100) is shown as having a driving device (120) directly, but the driving device (120) may be a separate device. For example, the electronic device (100) may be combined with a movable device such as a robot vacuum cleaner, and may be mounted on the robot vacuum cleaner to control the movement of the robot vacuum cleaner.

[0065] Meanwhile, the form of the illustrated electronic device is merely an example and can be implemented in various forms. Specific configurations constituting the electronic device (100) will be described later with reference to FIG. 3.

[0066] FIG. 3 is a block diagram illustrating the configuration of an electronic device according to at least one embodiment of the present disclosure.

[0067] Referring to FIG. 3, the electronic device (100) includes a camera (110), a driving device (120), a memory (130), and at least one processor (140).

[0068] According to one embodiment, the camera (110) is configured to capture a subject and obtain a captured image, wherein the image may include a static image or a dynamic image.

[0069] A camera (110) may be provided in at least one of the top area, bottom area, and side area of ​​the electronic device (100) to photograph a subject. The camera (110) may be implemented as a single camera, and may also be implemented as a plurality of cameras depending on the embodiment.

[0070] Additionally, the camera (110) may provide the captured image of the subject to the processor (140) in order to obtain identification information of the subject within the captured image.

[0071] Additionally, the camera (110) may be implemented as a wide-angle camera to capture a wide field of view, but is not limited thereto.

[0072] According to one example of the present disclosure, "shooting" may include the operation of an electronic device that controls a camera (e.g., a camera including an image sensor and a lens) equipped in an electronic device to convert an optical image formed through a lens into an electrical signal and acquire an image.

[0073] The camera (110) includes a lens, a shutter, an aperture, a solid-state image sensor, an AFE (Analog Front End), and a TG (Timing Generator). The shutter controls the time when light reflected from a subject enters the camera (110), and the aperture controls the amount of light incident on the lens by mechanically increasing or decreasing the size of the opening through which light enters. When light reflected from a subject accumulates as photocharge, the solid-state image sensor outputs an image based on the photocharge as an electrical signal. The TG outputs a timing signal for reading out pixel data from the solid-state image sensor, and the AFE samples and digitizes the electrical signal output from the solid-state image sensor.

[0074] According to one embodiment, the driving device (120) is configured to move the electronic device (100). To this end, the driving device (120) includes a motor, wheels, etc., and can move the electronic device (100) through the movement of the wheels.

[0075] Meanwhile, the driving device (120) can adjust the shooting direction of the electronic device (100). For example, the direction in which the camera (110) looks can be adjusted by adjusting the body position of the electronic device (100) as shown in FIG. 2, or the shooting direction can be adjusted by adjusting the lens position within the camera (110).

[0076] Here, the shooting direction refers to the direction in which the subject is photographed, and may also be referred to as the direction the electronic device faces, the projection direction, etc.

[0077] According to an embodiment, the memory (130) may store various programs, data, instructions, etc. used in the electronic device (100). In addition, the memory (130) may store various information according to various embodiments of the present disclosure.

[0078] The memory (130) may be implemented in the form of a memory embedded in the electronic device (100) or in the form of a memory that can be attached to and detached from the electronic device (100), depending on the purpose of data storage.

[0079] For example, data for driving the electronic device (100) may be stored in memory embedded in the electronic device (100), and data for the expansion function of the electronic device (100) may be stored in memory that is detachable from the electronic device (100).

[0080] In the case of memory embedded in an electronic device (100), it may be implemented in the form of volatile memory (e.g., DRAM (dynamic RAM), SRAM (static RAM), or SDRAM (synchronous dynamic RAM), etc.), non-volatile memory (e.g., OTPROM (one time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), etc.), hard drive, or solid state drive (SSD).

[0081] In the case of a memory that can be attached to and detached from an electronic device (100), it can be implemented in the form of a memory card (e.g., CF (compact flash), SD (secure digital), Micro-SD (micro secure digital), Mini-SD (mini secure digital), xD (extreme digital), MMC (multi-media card), etc.) or an external memory that can be connected to a USB port (e.g., USB memory).

[0082] The memory (130) may include various instructions required for the operation of the processor (140). Here, the instructions may include instructions for acquiring a captured image of a subject, instructions for controlling a driving device to move to a position corresponding to the direction of the subject's face based on the subject's body information, instructions for acquiring identification information of the subject based on body information and facial data, instructions for processing images, etc.

[0083] The memory (130) can store registration data or a database containing registration data.

[0084] Here, the registration data may refer to user data that has been previously registered in the electronic device (100) or server for a user using the electronic device. For example, the registration data may include physical information and facial data regarding the user.

[0085] Here, the registration data may be stored in a database stored on a server or in a database stored in the memory (130) of an electronic device (100).

[0086] According to an embodiment, at least one processor (140) controls the overall operation of the electronic device (100). Specifically, at least one processor (140) is connected to each component of the electronic device (100) and can control the overall operation of the electronic device (100).

[0087] At least one processor (140) can perform the operation of an electronic device (100) according to various embodiments by executing at least one instruction stored in memory.

[0088] According to an embodiment, at least one processor (140) may be implemented as a digital signal processor (DSP) that processes digital signals, a microprocessor, or a TCON (Timing controller). However, it is not limited thereto and may include or be defined by one or more of a central processing unit (CPU), a Micro Controller Unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a communication processor (CP), an ARM processor, or an AI (Artificial Intelligence) processor. Additionally, at least one processor (140) may be implemented as a System on Chip (SoC) or Large Scale Integration (LSI) with a built-in processing algorithm, or may be implemented in the form of a Field Programmable Gate Array (FPGA). At least one processor (140) can perform various functions by executing computer executable instructions stored in memory.

[0089] At least one processor (140) may include one or more of a CPU (Central Processing Unit), GPU (Graphics Processing Unit), APU (Accelerated Processing Unit), MIC (Many Integrated Core), DSP (Digital Signal Processor), NPU (Neural Processing Unit), hardware accelerator, or machine learning accelerator.

[0090] At least one processor (140) can obtain body information of a subject based on a first image obtained through a camera (110).

[0091] Here, "body information" may include data on the subject's physical characteristics (e.g., body proportions, height, etc.), data on the subject's gait (e.g., stride length, walking pose, etc.).

[0092] At least one processor (140) can control the driving device (120) so that the electronic device (100) moves to a position corresponding to the direction of the subject's face based on the subject's body information.

[0093] For example, at least one processor (140) can control the driving device (120) so that the electronic device (100) moves to a position corresponding to the direction of the subject's face when the similarity between the body information and the registration data exceeds a preset value.

[0094] At least one processor (140) can acquire facial data corresponding to a subject based on a second image acquired through a camera (110), and can acquire identification information of the subject based on body information and facial data.

[0095] Here, "second image" may refer to a captured image obtained by photographing the subject's face after obtaining a first image of the subject's body.

[0096] In this case, at least one processor (140) may obtain identification information corresponding to the subject based on weights for body information and facial data, respectively. This is explained in detail in FIG. 11.

[0097] At least one processor (140) can control one or any combination of other components of the electronic device and can perform operations or data processing related to communication. At least one processor (140) can execute one or more programs or instructions stored in memory. For example, at least one processor (140) can perform the method according to an embodiment of the present disclosure by executing one or more instructions stored in memory.

[0098] If the method according to the embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by a single processor or by a plurality of processors.

[0099] For example, when the first operation, the second operation, and the third operation are performed by the method according to the embodiment, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence dedicated processor).

[0100] One or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory (130). Alternatively, if one or more processors are dedicated artificial intelligence processors, the dedicated artificial intelligence processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model. The predefined operation rules or artificial intelligence models are characterized by being created through learning.

[0101] Here, "created through learning" means that a basic artificial intelligence model is trained using multiple learning data by a learning algorithm, thereby creating a predefined rule of operation or an artificial intelligence model configured to perform a desired characteristic (or objective). Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0102] An artificial intelligence model can be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through calculations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights can be updated during the learning process so that the loss or cost values ​​obtained by the artificial intelligence model are reduced or minimized.

[0103] Artificial neural networks may include deep neural networks (DNNs), such as, but are not limited to, Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), Generative Adversarial Networks (GANs), or Deep Q-Networks.

[0104] At least one processor (140) may be implemented as a single core processor including one core, or as one or more multicore processors including multiple cores (e.g., homogeneous multicore or heterogeneous multicore).

[0105] When at least one processor (140) is implemented as a multi-core processor, each of the multiple cores included in the multi-core processor may include internal processor memory such as cache memory or on-chip memory, and a common cache shared by multiple cores may be included in the multi-core processor.

[0106] Each of the multiple cores (or some of the multiple cores) included in the multi-core processor may independently read and execute program instructions for implementing the method according to the embodiment of the present disclosure, or all (or some of) of the multiple cores may be linked together to read and execute program instructions for implementing the method according to the embodiment of the present disclosure.

[0107] FIG. 4 is a detailed block diagram for illustrating an electronic device according to at least one embodiment of the present disclosure.

[0108] Referring to FIG. 4, an electronic device (100) according to one embodiment of the present disclosure may include a camera (110), a driving device (120), a memory (130), at least one processor (140), at least one sensor (150), and a communication unit (160). Parts that overlap with the above description will be omitted or abbreviated below.

[0109] At least one processor (140) can control the driving device to move to a position corresponding to the shooting of the subject when the subject is identified through at least one sensor (150).

[0110] For example, at least one processor (140) may move to the location where sound is detected and photograph the subject when sound is detected through a microphone.

[0111] At least one processor (140) may include a body detection module (141), a face detection module (142), an integration module (143), or an identification information determination module (144).

[0112] At least one processor (140) can detect the body of a subject through a body detection module (141) and obtain body information of the subject.

[0113] For example, at least one processor (140) can obtain body information of a subject based on a first image of the subject's body captured through a body detection module (141).

[0114] At least one processor (140) can detect the face of a subject through a face detection module (142), control a driving device (120) to move to a position corresponding to the direction of the subject's face, and acquire face data corresponding to the subject.

[0115] For example, at least one processor (140) can acquire face data corresponding to the subject based on a second image of the subject's face captured through a face detection module (142).

[0116] At least one processor (140) can generate weights for each of the body information and facial data through the integration module (143) and calculate an integrated similarity for the identified subject by applying the generated weights. This is explained in detail in FIG. 11.

[0117] Here, calculation refers to the process of processing or manipulating data. Particularly in the field of artificial intelligence, calculation can be used as a general term encompassing not only numerical, vector, and matrix operations, but also processes such as processing input data and optimizing weights.

[0118] Computation can refer to specific operations that process data to derive a result as part of an operation.

[0119] At least one processor (140) can obtain identification information for a user corresponding to registration data stored in a database through an identification information determination module (144) and determine it as identification information for an identified subject.

[0120] However, it is not limited to this, and at least one processor (140) may not obtain identification information for the subject if the integrated similarity is less than a preset value.

[0121] The operation of the body detection module (141), face detection module (142), integration module (143), or identification information determination module (144) is described in more detail in FIG. 5.

[0122] At least one sensor (150) is configured to identify a subject. The at least one sensor (150) may include at least one of an infrared sensor, a lidar sensor, an image sensor, an ultrasonic sensor, a vibration sensor, and a piezoelectric sensor.

[0123] For example, at least one processor (140) can identify a subject through a lidar sensor and can obtain not only distance information of the subject but also body information of the subject.

[0124] The communication unit (160) can communicate with an external server or an external electronic device. In particular, the communication unit (160) can receive information about a user from an external server or an external electronic device in order to obtain information about a user.

[0125] Additionally, the communication unit (160) may transmit information about the user obtained through the camera (110) or at least one sensor (150) to an external server or an external electronic device.

[0126] The communication unit (160) may include wired or wireless input / output interfaces (or input / output terminals) according to various standards. For example, one or more connection interfaces may include various interfaces such as HDMI (High Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB (D-subminiature), DVI (Digital Visual Interface), AP-based Wi-Fi (Wi-Fi, Wireless LAN Network), Bluetooth, Zigbee, wired / wireless LAN (Local Area Network), WAN (Wide Area Network), Ethernet, IEEE 1394, AES / EBU (Audio Engineering Society / European Broadcasting Union), Optical, Coaxial, etc.

[0127] An input interface (not shown) can receive various feedback from a user. For example, when an electronic device (100) receives user input regarding a setting that controls each configuration of the electronic device (100) through the input interface, it can operate each configuration in response to the user input.

[0128] For example, the input interface (180) may receive user input for generating weights for each of the body information and facial data. At least one processor (140) may generate weights for each of the body information and facial data based on the user input, and obtain identification information corresponding to the subject based on the weights for each of the body information and facial data.

[0129] Input interfaces may include, but are not limited to, a microphone, a touchscreen, etc., and may include various input interfaces capable of receiving user input.

[0130] Here, the microphone is a component for receiving sound input and converting it into an audio signal. The microphone is electrically connected to at least one processor (140) and can receive sound under the control of at least one processor (140).

[0131] For example, the microphone may be formed as an integrated unit on the upper side, front side, or other directions of the electronic device (100). Alternatively, the microphone may be provided in a remote control or the like, separate from the electronic device (100). In this case, the remote control may receive sound through the microphone and provide the received sound to the electronic device (100).

[0132] A microphone may include various components such as a microphone that collects analog sound, an amplifier circuit that amplifies the collected sound, an A / D converter circuit that samples the amplified sound and converts it into a digital signal, and a filter circuit that removes noise components from the converted digital signal.

[0133] Meanwhile, the microphone may be implemented in the form of a sound sensor, and any configuration capable of collecting sound is acceptable.

[0134] The operation of the electronic device (100) will be described in more detail below through FIGS. 5 to 12. FIGS. 5 to 12 describe individual embodiments for convenience of explanation. However, the individual embodiments of FIGS. 5 to 12 may be implemented in any combination.

[0135] FIG. 5 is a drawing for explaining the operation of a processor of an electronic device according to at least one embodiment of the present disclosure.

[0136] Referring to FIG. 5, the electronic device (100) can obtain identification information of a subject by processing a captured image obtained through a camera (110) through at least one processor (140) including a body detection module (141), a face detection module (142), an integration module (143), or an identification information determination module (144).

[0137] At least one processor (140) can calculate body similarity by extracting body information of a subject in a first image through a body detection module (141). For example, at least one processor (140) can calculate the degree of matching as a probability value by comparing the extracted body information, including body features and gait features of the subject, with the user's body information in a database.

[0138] At least one processor (140) can calculate facial similarity by extracting facial data corresponding to a subject in the second image through a facial detection module (142). For example, at least one processor (140) can calculate the degree of matching as a probability value by comparing the extracted facial data corresponding to the subject with the user's facial data in a database.

[0139] At least one processor (140) can generate weights for each of the body information and facial data through the integration module (143) and calculate the integrated similarity by applying the generated weights. For example, if at least one processor (140) sets the weight for facial data to 0.9 and the weight for body information to 0.1 through user input, it can calculate the integrated similarity as a probability value by applying the weights set for each of the body information and facial data.

[0140] At least one processor (140) can determine whether there is a match with the user's identification information stored in the database based on the integrated similarity through the identification information determination module (144).

[0141] For example, at least one processor (140) can obtain the identification information of a user stored in a database as the identification information of a subject if the integrated similarity exceeds a preset value. On the other hand, at least one processor (140) may not obtain the identification information of a subject if the integrated similarity does not exceed a preset value.

[0142] However, it is not limited to this, and at least one processor (140) may obtain information regarding whether there is a match for a user stored in a database based on integrated similarity.

[0143] FIG. 6 is a drawing for explaining the operation of an electronic device according to at least one embodiment of the present disclosure to identify and move a subject.

[0144] Referring to FIG. 6, the electronic device (100) may first acquire body information of the subject and then determine whether the electronic device (100) drives based on whether the body similarity exceeds a preset value.

[0145] For example, when body information in a captured image of a specific object (11) among a plurality of objects (11, 12, 13) is compared with registered data stored in a database and it is identified that it has a similarity exceeding a preset value, it can be moved to a position corresponding to the direction of the subject's face (10) in order to obtain a captured image of the subject's face (11').

[0146] Here, the position corresponding to the direction of the subject's face (10) may be determined based on the distance between the subject (10) and the electronic device (100) and the direction of the subject's face (10).

[0147] For example, the electronic device (100) may move to a position close to the subject (10) and where the direction of the subject's face and the direction of the electronic device's camera scan match in order to acquire a captured image of the subject's (10) face.

[0148] On the other hand, if the calculated body similarity does not exceed a preset value, the electronic device (100) may not move to a position corresponding to the direction of the subject's (10) face.

[0149] FIG. 7 is a flowchart illustrating the overall operation of an electronic device according to at least one embodiment of the present disclosure.

[0150] Referring to FIG. 7, the electronic device (100) can calculate body similarity and facial similarity for a subject in stages during autonomous driving to obtain identification information of the subject.

[0151] For example, the electronic device (100) can perform autonomous driving (710) for object recognition.

[0152] The electronic device (100) may identify whether the identification target is recognized (720), and if the identification target is not recognized, it may terminate autonomous driving (721).

[0153] When an identification target is recognized, the electronic device (100) can acquire a body image of the subject being identified, perform data preprocessing, and extract body similarity (730).

[0154] The electronic device (100) identifies whether the extracted body similarity exceeds a threshold value (740), and if the body similarity exceeds the threshold value, it can move toward the subject's face (750) to obtain a face image of the subject.

[0155] The electronic device (100) can acquire a facial image of a subject, perform data preprocessing, and extract facial similarity (760).

[0156] The electronic device (100) can apply weights to the extracted body similarity and facial similarity and calculate the integrated similarity (770).

[0157] The electronic device (100) identifies whether the calculated integrated similarity exceeds a threshold value (780), and if the integrated similarity exceeds the threshold value, it can obtain identification information of the subject (790) based on user information stored in the database.

[0158] FIG. 8 is a flowchart illustrating the operation of an electronic device according to at least one embodiment of the present disclosure to additionally learn facial data.

[0159] Referring to FIG. 8, the electronic device (100) can additionally learn facial data about the subject to update user information stored in the database.

[0160] For example, an electronic device (100) can recognize a user (810) based on an existing Face ID. Here, "existing Face ID" may mean identification information about a user's face stored in a database.

[0161] The electronic device (100) can identify whether additional learning (820) is required for facial data stored in a database. For example, the electronic device (100) may identify that additional learning is required for facial data if it identifies that a preset period has elapsed since the last date the facial data stored in the database was learned, or if it identifies that there has been a change in the user's facial shape.

[0162] However, it is not limited to this, and the electronic device (100) may receive a signal from an external server or external electronic device through a communication unit that additional learning of facial data is required.

[0163] If the electronic device (100) is identified as needing additional learning of facial data, it can move (830) so that the subject's face is recognized.

[0164] The electronic device (100) can extract facial data (840) based on a captured image of the subject's face and map the extracted facial data to each user and store it in a database (DB) that stores user information (850).

[0165] The electronic device (100) may also identify (910) whether additional learning of body information is required after completing additional learning of facial data. This is explained in detail in FIG. 9.

[0166] FIG. 9 is a flowchart illustrating the operation of an electronic device according to at least one embodiment of the present disclosure to additionally learn body data.

[0167] Referring to FIG. 9, the electronic device (100) can additionally learn body data about the subject to update user information stored in the database.

[0168] The electronic device (100) can identify (910) whether additional learning is required for body information stored in a database. For example, the electronic device (100) may identify that additional learning is required for body information if the user recognition rate determined based on the body information stored in the database is less than a preset value (e.g., if the accuracy of body judgment is 0.7 or less).

[0169] However, it is not limited to this, and the electronic device (100) may receive a signal from an external server or external electronic device through a communication unit that additional learning about body information is required.

[0170] On the other hand, if the user recognition rate determined based on the body information stored in the database exceeds a preset value, the electronic device (100) may identify that no additional learning of the body information is required and complete the learning (911).

[0171] If the electronic device (100) identifies that additional learning of body information is required, it can track and move (920) the recognized user while driving.

[0172] The electronic device (100) can extract body information based on an image of the user and perform data preprocessing (930).

[0173] The electronic device (100) can extract the user's physical characteristics or gait characteristics (940) through data preprocessing.

[0174] The electronic device (100) can map the extracted user's physical characteristics or gait characteristics to each user and store them (950) in a database (DB) where user information is stored.

[0175] FIG. 10 is a flowchart illustrating the operation of an electronic device according to at least one embodiment of the present disclosure identifying a subject based on the subject's body information.

[0176] Referring to FIG. 10, the electronic device (100) can extract body similarity based on static data and dynamic data.

[0177] For example, an electronic device (100) can recognize a subject (1010) through at least one sensor or camera.

[0178] The electronic device (100) can acquire a body image of a subject to be identified, perform data preprocessing, and extract body similarity (1020).

[0179] The electronic device (100) can extract body similarity (1021) based on static data. For example, the electronic device (100) can photograph a subject to obtain static data containing data on the subject's body features, and calculate body similarity by comparing the obtained static data with user information stored in a database.

[0180] The electronic device (100) can extract body similarity (1022) based on dynamic data. For example, the electronic device (100) can photograph a subject to obtain dynamic data including data on the subject's walking characteristics, and calculate body similarity by comparing the obtained dynamic data with user information stored in a database.

[0181] The electronic device (100) identifies whether the extracted body similarity exceeds a threshold (1030), and if the body similarity exceeds the threshold, it can move toward the subject's face (1040) so that the subject's face is recognized.

[0182] The electronic device (100) can acquire a facial image of a subject, perform data preprocessing, and extract facial similarity (1050).

[0183] The electronic device (100) can apply weights to the extracted body similarity and facial similarity and calculate the integrated similarity (1060).

[0184] The electronic device (100) identifies whether the calculated integrated similarity exceeds a threshold value (1070), and if the integrated similarity exceeds the threshold value, it can identify that the information about the identified subject matches the user's information stored in the database (1090).

[0185] On the other hand, if the integrated similarity does not exceed a threshold, the electronic device (100) can identify that the information about the identified subject is inconsistent (1080) with the user's information stored in the database.

[0186] Meanwhile, if the electronic device (100) identifies that the information about the identified subject matches the user information stored in the database (1090), it can obtain the identification information of the subject based on the user information stored in the database.

[0187] FIG. 11 is a table for explaining the operation of an electronic device according to at least one embodiment of the present disclosure applying weights to data.

[0188] The electronic device (100) can generate weights for each of the body information and facial data, and obtain identification information corresponding to the subject based on the weights for each of the body information and facial data.

[0189] Referring to the table (1100) in Fig. 11, weights can be applied differently depending on the degree of recognition of the subject's face and body, and fixed weight values ​​can also be applied according to preset values.

[0190] For example, if the weight for facial data is set to 0.9 and the weight for body information is set to 0.1 in advance in the electronic device (100) (1110), the integrated similarity can be calculated as a probability value by applying the weights set for each of the body information and facial data.

[0191] However, it is not limited to this, and weights may be generated for each of the body information and facial data based on user input, and identification information corresponding to the subject may be obtained based on the weights for each of the body information and facial data.

[0192] The electronic device (100) may obtain identification information of the subject by applying not only fixed weights but also weights generated based on the recognition rate.

[0193] For example, the electronic device (100) may obtain weights for body information and facial data, respectively, based on a first recognition rate obtained from a first image and a second recognition rate obtained from a second image, and may obtain identification information corresponding to a subject based on the weights for each of the obtained body information and facial data.

[0194] Specifically, referring to the table (1100) in FIG. 11, the electronic device (100) may calculate the integrated similarity as a probability value by applying a weight of 0 to the face data and a weight of 1 to the body information when the face of the subject is not recognized and only the body is recognized (1120).

[0195] Here, "first recognition rate" may mean that it corresponds to the recognition rate for the subject's body, and "second recognition rate" may mean that it corresponds to the recognition rate for the subject's face.

[0196] The electronic device (100) may obtain a first recognition rate and a second recognition rate based on at least one of the angle of the camera, the posture of the subject, the illumination of the space where the electronic device is located, and the distance between the electronic device and the subject.

[0197] For example, when the subject is sitting, the electronic device (100) can apply a higher weight to the face because the recognition rate for the face is calculated to be higher than that for the body.

[0198] Specifically, referring to the table (1100) in FIG. 11, the electronic device (100) may calculate the integrated similarity by applying a higher weight to the face when the subject is sitting facing the electronic device (1150) than when the subject is standing facing the electronic device (1140).

[0199] If the first recognition rate is greater than the second recognition rate, the electronic device (100) may obtain a greater weight for the body information among the body information and facial data.

[0200] Specifically, referring to the table (1100) in FIG. 11, the electronic device (100) corresponds to a case where the recognition rate for the body is greater than the recognition rate for the face when the subject walks around the electronic device (100) (1130), so the integrated similarity may be calculated as a probability value by applying a weight of 0.3 for the face data and a weight of 0.7 for the body information.

[0201] Meanwhile, combined similarity can refer to the probability value obtained by multiplying facial similarity and body similarity by their respective weights and adding them together. For example, combined similarity may be calculated using the following formula.

[0202] Combined Similarity = Face Weight × Face Similarity + Body Weight × Body Similarity

[0203] Meanwhile, the closer the integrated similarity is to 1 rather than 0, the higher the accuracy of the identification information obtained for the identified subject.

[0204] FIG. 12 is a flowchart illustrating the operation of an electronic device acquiring identification information according to at least one embodiment of the present disclosure.

[0205] The electronic device can acquire body information of a subject based on a first image of the subject's body (S1210). Here, "body information" may mean information about the subject's body including static data or dynamic data. "Static data" may correspond to data related to the subject's body. "Dynamic data" may correspond to data regarding the subject's steps.

[0206] The electronic device can move to a position corresponding to the direction of the subject's face based on the acquired body information of the subject (S1220). Here, the case where the electronic device (100) moves to a position corresponding to the direction of the subject's face based on the body information may include a case where the similarity between the body information and the registered data stored in the database exceeds a preset value.

[0207] The electronic device can acquire facial data corresponding to the subject based on a second image of the subject's face (S1230). Here, "second image" may refer to a captured image obtained by capturing the subject's face after acquiring a first image of the subject's body.

[0208] The electronic device can obtain identification information of a subject based on acquired body information and facial data (S1240). Here, "identification information of a subject" may refer to unique personal information that can identify the subject. For example, the identification information of a subject may include the subject's physical characteristics, the subject's personal information (e.g., name, date of birth, etc.), etc.

[0209] The method for obtaining identification information described in FIG. 12 may be performed by a device having various configurations such as FIG. 3 and FIG. 4 described above, but is not necessarily limited thereto and may be performed by a device having various configurations.

[0210] The various embodiments described above may be implemented as individual embodiments, or at least one embodiment may be combined with one another, either wholly or partially, to be implemented together in a single device.

[0211] According to the various embodiments described above, by making a judgment based on facial data after making a judgment based on body information regarding the subject, the efficiency and accuracy of subject recognition can be improved, and consistent performance can be maintained in response to various environments, such as outdoors. Ultimately, the user experience can be improved.

[0212] Various embodiments of the present disclosure may be implemented as software stored on a machine-readable storage media that can be mounted on or connected to a smartphone, a user terminal device, and various other electronic devices (e.g., a computer).

[0213] Specifically, a non-transient readable storage medium may be provided that stores software for sequentially performing the steps of: acquiring body information of a subject based on a first image acquired through an electronic device; moving the electronic device to a position corresponding to the direction of the subject's face based on the body information; acquiring facial data corresponding to the subject based on a second image acquired through an electronic device; and acquiring identification information of the subject based on the body information and facial data.

[0214] A device equipped with such a non-transient readable medium can perform various operations, such as acquiring body information of a subject based on the captured images described in the various embodiments above, controlling a driving device to move an electronic device to a position corresponding to the direction of the subject's face, acquiring face data corresponding to the subject based on the captured images, and acquiring identification information of the subject based on the body information and face data.

[0215] In non-transitory readable storage media, 'non-transitory' simply means that the storage medium does not contain a signal and is tangible; it does not distinguish whether data is stored semi-permanently or temporarily on the storage medium.

[0216] Alternatively, a program for performing the methods according to the various embodiments described above may be distributed online through an application store. In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a storage medium such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0217] Each component (e.g., module or program) according to various embodiments may consist of a single or multiple entities, and some of the aforementioned sub-components may be omitted, or other sub-components may be further included in various embodiments. Generally or additionally, some components (e.g., module or program) may be integrated into a single entity to perform the same or similar functions as those performed by each of the respective components prior to integration.

[0218] Operations performed by a module, program, or other component according to various embodiments may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0219] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

1. In an electronic device, camera; Driving device; Memory for storing at least one instruction; and It includes at least one processor that executes the above at least one instruction; and The above-mentioned at least one processor is, Based on the first image obtained through the above camera, body information corresponding to the subject is obtained, and Based on the above body information, the driving device is controlled to move to a position corresponding to the direction of the subject's face, and Based on a second image obtained through the camera, facial data corresponding to the subject is obtained, and An electronic device that obtains identification information corresponding to the subject based on the above body information and the above facial data.

2. In Paragraph 1, The above-mentioned at least one processor is, An electronic device that controls the driving device to move to a position corresponding to the direction of the subject's face when the similarity between the above body information and the registration data exceeds a preset value.

3. In Paragraph 1, The above-mentioned at least one processor is, An electronic device that obtains identification information corresponding to the subject based on weights for each of the above body information and above facial data.

4. In Paragraph 1, The above-mentioned at least one processor is, Based on the first recognition rate obtained from the first image and the second recognition rate obtained from the second image, weights for each of the body information and the facial data are obtained, and An electronic device that acquires identification information corresponding to the subject based on weights for each of the acquired body information and facial data.

5. In Paragraph 4, The above-mentioned at least one processor is, The first recognition rate and the second recognition rate are obtained based on at least one of the angle of the camera, the posture of the subject, the illumination of the space where the electronic device is located, and the distance between the electronic device and the subject. The above first recognition rate is, Corresponding to the recognition rate of the body of the above subject, The above second recognition rate is, An electronic device corresponding to the recognition rate of the face of the above-mentioned subject.

6. In Paragraph 4, The above-mentioned at least one processor is, An electronic device that obtains a greater weight for the body information among the body information and the facial data when the first recognition rate is greater than the second recognition rate.

7. In Paragraph 1, The above-mentioned at least one processor is, Based on user input, weights are obtained for each of the above body information and above facial data, and An electronic device that obtains identification information corresponding to the subject based on weights for each of the above body information and above facial data.

8. In Paragraph 1, The above body information is, An electronic device comprising static data and dynamic data of the above-mentioned subject.

9. In Paragraph 8, The above static data is, Corresponding to data related to the body of the above-mentioned subject, and The above dynamic data is, An electronic device corresponding to data related to the steps of the above-mentioned subject.

10. In Paragraph 1, further including at least one sensor, The above-mentioned at least one processor is, When the subject is identified through the at least one sensor, the driving device is controlled to move to a position corresponding to the shooting of the subject, and The above-mentioned at least one sensor is, An electronic device comprising at least one of an infrared sensor, a lidar sensor, an image sensor, an ultrasonic sensor, a vibration sensor, and a piezoelectric sensor.

11. In a method for controlling an electronic device, A step of obtaining body information corresponding to a subject based on a first image obtained through the electronic device; A step of moving to a position corresponding to the direction of the subject's face based on the above body information; A step of acquiring facial data corresponding to the subject based on a second image acquired through the electronic device; and A control method comprising the step of obtaining identification information of the subject based on the above body information and the above facial data.

12. In Paragraph 11, The step of moving to a position corresponding to the direction of the face of the subject is, A control method comprising: a step of identifying that driving of the electronic device is required when the similarity between the above body information and the registration data exceeds a preset value.

13. In Paragraph 11, The step of obtaining identification information of the above subject is, A step of corresponding to the subject based on weights for each of the above body information and above facial data; A control method including 14. In Paragraph 11, The step of obtaining identification information of the above subject is, Based on the first recognition rate obtained from the first image and the second recognition rate obtained from the second image, weights for each of the body information and the facial data are obtained, and A control method comprising the step of obtaining identification information corresponding to the subject based on weights for each of the body information and facial data obtained above.

15. A non-transient readable recording medium comprising a program for executing a method of controlling an electronic device, The control method of the above electronic device is, A step of obtaining body information corresponding to a subject based on a first image obtained through the electronic device; A step of moving to a position corresponding to the direction of the subject's face based on the above body information; A step of acquiring facial data corresponding to the subject based on a second image acquired through the electronic device; and A non-transient readable recording medium comprising the step of obtaining identification information of the subject based on the above body information and the above facial data.

Images