Method, electronic device, program, and storage medium for determining, by using object, activity to be performed by user

Abstract

According to one embodiment, a method for inferring, by using an object, an activity to be performed by a user can be provided. The method may comprise the operations: of acquiring, through an electronic device, an image obtained by photographing a hand of a user and an object gripped by the hand; on the basis of an attribute of the object and the form of the hand for gripping the object in the image, acquiring, by using the object, activity information related to at least one activity to be performed by the user; identifying at least one application which corresponds to the at least one activity and which is executable in the electronic device; and executing, in the electronic device, an application included in the at least one identified application.

Description

Method for determining an activity to be performed by a user using an object, electronic device, program, and storage medium

[0001] The present disclosure relates to a method, electronic device, program, and storage medium for determining an activity to be performed by a user using an object, and more specifically, to determining an activity to be performed by a user using an object and performing an operation corresponding to the determined activity on an electronic device.

[0002] Recently, wearable electronic devices that provide extended reality (XR) services, including augmented reality (AR), virtual reality (VR), or mixed reality (MR), are being developed. For example, users can enjoy various XR services such as cameras, games, video streaming, or navigation while wearing a head-mounted display (HMD) type wearable electronic device on their head or XR glasses on their face.

[0003] Hand tracking, eye tracking, or motion controllers are primarily used as input methods for wearable electronic devices that provide XR services. For example, to run an application in a 3D XR space provided by a wearable electronic device, the user can blink while keeping their gaze fixed on an application icon on the home screen displayed in the 3D XR space, clench their hand while pointing it toward the application, or press a button on a motion controller while pointing the motion controller held in their hand toward the application.

[0004] Recently, artificial intelligence systems capable of achieving human-level intelligence are being utilized in various fields. Unlike conventional rule-based smart systems, artificial intelligence systems are systems in which machines learn, make judgments, and become smarter on their own. As artificial intelligence systems improve in recognition accuracy and gain a more accurate understanding of user preferences with continued use, existing rule-based smart systems are gradually being replaced by deep learning-based artificial intelligence systems.

[0005] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art in relation to the present disclosure.

[0006] According to one embodiment, a method for determining an activity to be performed by a user using an object may be provided. The method may include the operation of acquiring an image of a user's hand and an object held by the hand through an electronic device. The method may include the operation of acquiring activity information regarding at least one activity to be performed by the user using the object, based on the attributes of the object and the shape of the hand for holding the object in the image. The method may include the operation of identifying at least one application that corresponds to the at least one activity and is executable on the electronic device, based on the activity information. The method may include the operation of executing an application included in the identified at least one application on the electronic device.

[0007] According to one embodiment, an electronic device may be provided comprising one or more processors including memory for storing instructions and processing circuitry. When the instructions are executed individually or collectively by the one or more processors, the electronic device may acquire through the electronic device an image of a user's hand and an object held by the hand. When the instructions are executed by the one or more processors, the electronic device may acquire activity information regarding at least one activity to be performed by the user using the object, based on the attributes of the object in the image and the shape of the hand for holding the object. When the instructions are executed by the one or more processors, the electronic device may identify at least one application that corresponds to the at least one activity and is executable on the electronic device, based on the activity information. When the instructions are executed by the one or more processors, the electronic device may execute an application included in the identified at least one application on the electronic device.

[0008] A computer-readable non-transitory recording medium according to one embodiment of the present invention may store at least one instruction and / or instruction that causes an electronic device to perform the method or operation of the electronic device described above when executed.

[0009] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0010] FIG. 1 is a flowchart of a method for determining an activity to be performed by a user using an object, according to one embodiment.

[0011] FIG. 2 is a diagram illustrating a method for determining an activity to be performed using a user's right hand with an object, according to one embodiment.

[0012] FIG. 3 is a diagram illustrating a method for determining an activity to be performed after a user changes their posture, according to one embodiment.

[0013] FIG. 4a is a diagram illustrating a method for obtaining activity information regarding an activity to be performed by a user using an object, according to one embodiment.

[0014] FIG. 4b is a diagram illustrating a method for recommending an action to be performed in an electronic device based on determined activity information according to one embodiment.

[0015] FIG. 5a is a diagram illustrating a method for determining an activity to be performed using two objects with both hands of a user, according to one embodiment.

[0016] FIG. 5b is a drawing for explaining a method of displaying an image according to an activity to be performed using two objects with both hands of a user, according to one embodiment.

[0017] FIG. 6 is a diagram illustrating a method for determining an activity to be performed using both hands of a user with a single object, according to one embodiment.

[0018] FIG. 7 is a diagram illustrating a method for determining an activity to be performed by a user using two or more objects according to one embodiment.

[0019] FIG. 8 is a block diagram of a wearable electronic device according to one embodiment.

[0020] FIG. 9 is a block diagram of an electronic device in a network environment according to one embodiment.

[0021] Embodiments of the present disclosure are described below in detail with reference to the attached drawings so that those skilled in the art can easily implement them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present disclosure in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0022] The terms used in this disclosure are described in their current, general form considering the functions mentioned herein; however, they may refer to various other terms depending on the intent of those skilled in the art, case law, or the emergence of new technologies. Accordingly, the terms used in this disclosure should not be interpreted solely by their names, but should be interpreted based on the meaning of the terms and the overall content of this disclosure.

[0023] Additionally, terms such as the first, second, third, ..., Nth may be used to describe various components, but the components should not be limited by these terms. These terms are used for the purpose of distinguishing one component from another.

[0024] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are "directly connected," but also cases where they are "electrically connected" with other components interposed between them. Furthermore, when a part is described as "including" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0025] Phrases such as "in one embodiment" appearing in various places in this disclosure do not necessarily refer to the same embodiment.

[0026] One embodiment of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented by various numbers of hardware and / or software configurations that execute specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors or by circuit configurations for a specific function. Additionally, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented as algorithms executed on one or more processors. Furthermore, the present disclosure may employ prior art for electronic configuration, signal processing, and / or data processing. Terms such as "mechanism," "element," "means," and "configuration" may be used broadly and are not limited to mechanical and physical configurations.

[0027] Furthermore, the connecting lines or connecting members between the components depicted in the drawings are merely illustrative of functional connections and / or physical or circuit connections. In the actual device, connections between components may be represented by various alternative or added functional connections, physical connections, or circuit connections.

[0028] Artificial intelligence technology consists of machine learning (e.g., deep learning) and component technologies utilizing machine learning. Machine learning is an algorithmic technology that classifies and learns the characteristics of input data on its own, and component technologies are technologies that mimic functions such as cognition and judgment of the human brain by utilizing deep learning machine learning algorithms, and consist of technology fields such as linguistic understanding, visual understanding, reasoning / prediction, knowledge representation, and motion control.

[0029] The various fields where artificial intelligence technology is applied are as follows. Linguistic understanding is a technology that recognizes, applies, and processes human language and text, and includes natural language processing, machine translation, dialogue systems, question answering, and speech recognition / synthesis. Visual understanding is a technology that perceives and processes objects like human vision, and includes object recognition, object tracking, image search, person recognition, scene understanding, spatial understanding, and image enhancement. Inference and prediction is a technology that judges information to logically infer and predict, and includes knowledge / probability-based inference, optimization prediction, preference-based planning, and recommendation. Knowledge representation is a technology that automatically processes human experiential information into knowledge data, and includes knowledge construction (data generation / classification) and knowledge management (data utilization). Motion control is a technology that controls the autonomous driving of vehicles and the movement of robots, and includes motion control (navigation, collision, driving) and manipulation control (behavior control).

[0030] The functions related to artificial intelligence according to the present disclosure are operated through a processor and memory. The processor may be composed of one or more processors. In this case, the one or more processors may be general-purpose processors such as CPUs, APs, and DSPs (Digital Signal Processors), graphics-dedicated processors such as GPUs and VPUs (Vision Processing Units), or artificial intelligence-dedicated processors such as NPUs. The one or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if the one or more processors are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0031] The predefined rules of operation or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that a predefined rules of operation or artificial intelligence models configured to perform a desired characteristic (or objective) are created by a basic artificial intelligence model being trained using multiple learning data by a learning algorithm. Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0032] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through operations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. Artificial neural networks may include deep neural networks (DNNs), such as Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), or Deep Q-Networks, but are not limited to the examples mentioned above.

[0033] The three-dimensional XR space provided by an electronic device in the present disclosure may be implemented by displaying a virtual image containing virtual objects, but is not limited thereto. For example, the three-dimensional XR space may be implemented by displaying a combination of a real image and a virtual image of a real scene containing real-world objects. For example, the three-dimensional XR space may be implemented by overlaying a virtual image onto a physical environment space of the real world that is viewed through the lens of the electronic device.

[0034] In this disclosure, an object refers to any physical object capable of interacting with a user, and, for example, an object may be a tool, equipment, a musical instrument, or a material, but is not limited thereto. An object may be an object that a user can interact with using their hand. For example, an object may be an object that a user can grasp with their hand, wear on the hand, attach to the hand, or wrap around the hand. An object may be an object that is not commutatively connected to the user's electronic device. For example, an object may be a non-electronic device that does not include electronic circuits. In this disclosure, the attributes of an object are information that can be analyzed from an image, and may include, for example, the type, size, volume, shape, color, texture, material, transparency, location, orientation, distance, use, operating state, and interaction with other objects (e.g., the user's hand), but is not limited thereto. The attributes of an object may be expressed as object information regarding the object.

[0035] In the present disclosure, the interaction between an object and a user refers to the manner in which the user utilizes the object, and the interaction may include, but is not limited to, the form or posture in which the user grasps the object with their hand. For example, the interaction may include the manner in which the object is worn or attached to the user's hand.

[0036] In the present disclosure, interaction information represents an interaction between an object and a user, and, for example, the interaction information may include, but is not limited to, object information regarding the object, posture information regarding the shape or posture of a hand for grasping the object, wearing information regarding the manner in which the object is worn or attached to the hand, and motion information regarding the movement of the hand, and may include various information for describing the interaction between the object and the user.

[0037] In the present disclosure, "activity" refers to a series of actions that can be performed by a user, and, for example, an activity may be exercise, playing a musical instrument, cooking, drawing, writing, or knitting, but is not limited thereto. An activity may be performed by the user using an object or may be performed by the user based on interaction between the user and an object.

[0038] In the present disclosure, activity information represents an activity performed or to be performed by a user, for example, activity information may represent an action performed or to be performed by a user using an object, or an action performed or to be performed by a user based on interaction between a user and an object.

[0039] In the present disclosure, the environment refers to a place where the user is located, and environmental information regarding the environment may include, for example, the location of the place where the user is located, whether the user is located indoors or outdoors, whether there are other people around the user, or the temperature around the user, but is not limited thereto.

[0040] In the present disclosure, history refers to the history of a user's use of an electronic device and may be stored in the form of history information. In the present disclosure, history information may include behavior data that tracks how a user behaves on a specific platform (application or webpage), such as content input, function usage records, search history, webpage visit history, activity history, exercise history, and viewing history.

[0041] When using hand tracking or motion controllers for input on wearable electronic devices, such as a head-mounted display (HMD), users can point their hand or motion controller toward a specific part of the 3D XR space to make a selection. This method is inconvenient for the user because it requires larger movements compared to mouse operations in a desktop environment or touch operations on a smartphone. Furthermore, in a 3D XR space where the user's location or separate tools can be utilized, the continuity of the experience is degraded because the user must put down the tool they were using, point their hand or the hand holding the motion controller toward—for example, an application—to make a selection, and then pick up the tool again. For example, a user intending to display sheet music and play an instrument with both hands on a wearable electronic device utilizing hand tracking or motion controllers must first set the wearable electronic device to a mode suitable for the performance environment and select the application for displaying the sheet music using their hand or motion controller before grasping the instrument with both hands. On the other hand, according to one embodiment of the present disclosure, simply by a user wearing a wearable electronic device assuming a posture of holding a musical instrument with both hands, the wearable electronic device can be set to a mode suitable for a performance environment and an application for displaying sheet music can be executed, which can improve the continuity of the user experience.

[0042] The present disclosure will be described in detail below with reference to the attached drawings.

[0043] FIG. 1 is a flowchart of a method for determining an activity to be performed by a user using an object, according to one embodiment.

[0044] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0045] According to one embodiment, the following operations 110, 120, 130, and 140 may be understood to be performed by a processor (e.g., processor (820) of FIG. 8 and processor (920) of FIG. 9) of an electronic device (e.g., electronic device (801) of FIG. 8 and electronic device (901) of FIG. 9).

[0046] According to one embodiment, an image of the user may be obtained in operation 110. The image may be obtained by photographing the user's hand and objects around the hand through an electronic device. The electronic device may be a wearable electronic device, and the wearable electronic device may be, for example, a head-mounted display (HMD) worn on the head or XR glasses worn on the face, but is not limited thereto. The electronic device may be a smartphone, a tablet, or a computing device. The image may be obtained by photographing the user's hand and objects around the hand through a camera and a depth sensor of the electronic device, and the image may include depth information obtained through the depth sensor. The depth sensor may be an infrared sensor, a Time-of-Flight (ToF) sensor, an ultrasonic sensor, a light detection and ranging (LiDAR) sensor, or a stereo camera, and depth information may be generated by the depth sensor.

[0047] According to one embodiment, the user image may include the user's hand and objects around the hand. For example, the user image may include the user's hand and objects held by the hand, but is not limited thereto. For example, the user image may include objects in front of the user's hand, objects behind the user's hand, objects beyond the user's hand, objects next to the user's hand, or objects covering the user's hand. Objects around the user's hand may be tools, equipment, instruments, or materials capable of interacting with the user, but are not limited thereto. Objects capable of interacting with the user may be held by the user's hand, or worn or attached to the user's hand. Objects may be non-electronic devices that are not commutatively connected to the user's electronic device.

[0048] According to one embodiment, an image including a user's hand and objects around the hand may include depth information to more effectively identify the posture of the hand and the type of objects.

[0049] According to one embodiment, a wearable electronic device worn by a user on the head or face can capture an image of the user's hand and objects around the hand as viewed from the user's head height. Refer to FIGS. 2, FIGS. 3, FIGS. 5a, FIGS. 5b, FIGS. 6, and FIGS. 7 for an image of the user's hand and objects around the hand as viewed from the user's head height.

[0050] According to one embodiment, a user may look at the front display and front camera of an electronic device, such as a smartphone, tablet, or computing device, which is mounted or placed at a suitable height and at a suitable distance from the user, and the image captured by the electronic device may include the user's hand and objects around the hand viewed from the front of the user. A foldable smartphone may adjust the folded angle of the screen while placed with a portion of the screen folded so that the user's hand and objects around the hand are included within the field of view. Images of the user's hand and objects around the hand taken at head height by a wearable electronic device worn on the user's head or face, and images of the user's hand and objects around the hand taken from the front through the front camera of an electronic device, such as a smartphone, tablet, or computing device, may be used to individually train different artificial intelligence models, but are not limited thereto, and may also be used to train the same artificial intelligence model.

[0051] According to one embodiment, an activity to be performed by a user in operation 120 can be determined. The activity to be performed by the user can be inferred based on the attributes of an object in an image and the shape of a hand. For example, the activity to be performed by the user can be determined based on the attributes of an object in an image and the shape of a hand for grasping the object. According to one embodiment, the activity to be performed by the user can be inferred by applying an image obtained in operation 110 to an artificial intelligence model. The artificial intelligence model receives the image obtained in operation 110 as input and can generate activity information regarding the activity to be performed by the user as output. The process of generating activity information by applying an image including the user's hand and objects around the hand to an artificial intelligence model to infer the activity to be performed by the user will be described later with reference to FIG. 4a.

[0052] According to one embodiment, an artificial intelligence model may be pre-trained to generate activity information regarding an activity to be performed by a user by learning images of multiple users' hands and objects around the hands. The artificial intelligence model may include, but is not limited to, a CNN. According to one embodiment, the artificial intelligence model may learn images of a user's hand and objects around the hands of an electronic device, and actions selected or executed by the user on the electronic device within a set time after the images are taken, thereby enabling the artificial intelligence model to provide personalized results to the user.

[0053] According to one embodiment, an action that can be performed on an electronic device in action 130 can be identified. The electronic device corresponds to an activity determined to be performed by a user in action 120, and can identify an action that can be performed on the electronic device.

[0054] According to one embodiment, an action executable on an electronic device may be, but is not limited to, the installation and / or execution of an application corresponding to a determined activity. For example, an action executable on an electronic device may include the playback of multimedia content corresponding to a determined activity, the change of a setting value of the electronic device (e.g., setting or disabling Do Not Disturb mode, setting or disabling notifications, changing the wallpaper, setting or disabling exercise mode, setting or disabling drawing mode, setting or disabling instrument playing mode, and / or setting or disabling cooking mode), the transmission of a message, and a phone connection. An action executable on an electronic device may refer to an action executable at the operating system level of the electronic device. The process of identifying an action executable on an electronic device corresponding to a determined activity will be described later with reference to FIG. 4b.

[0055] According to one embodiment, an operation may be performed on an electronic device in operation 140. The operation performed may be the operation identified in operation 130.

[0056] According to one embodiment, if one action that can be executed on an electronic device is identified in action 130, that action may be executed immediately on the electronic device, but is not limited thereto. For example, a countdown may be started with a message indicating that the action will be executed, and the action may be executed if no user intervenes before the countdown is completed.

[0057] According to one embodiment, when a plurality of actions executable in an electronic device are identified, those actions may be executed together immediately in the electronic device. For example, the identified actions may be executed sequentially or in parallel in the electronic device. Actions executed sequentially or in parallel may be compatible with each other. For example, actions that run in the background or run in picture-in-picture (PIP) mode may be executed together with other actions.

[0058] According to one embodiment, a plurality of actions executable on an electronic device are identified, and the identified actions may be incompatible with one another. For example, setting to a first background and setting to a second background are incompatible with each other, and immersive applications cannot be executed simultaneously. In such cases, an action with a higher priority (e.g., an action more associated with an activity determined to be performed by the user) may be executed, or a countdown may begin in order of priority starting from the action with the highest priority, and if the user does not make a selection before the countdown is completed, the action with the highest priority may be executed. An immersive application refers to an application executed throughout the entire three-dimensional XR space.

[0059] According to one embodiment, a plurality of actionable operations in an electronic device are identified, and a list containing the identified operations may be provided to a user through the electronic device as a user interface. The actionable operations included in the list may include operations that have no history of being executed in the electronic device. According to one embodiment, a list containing the identified actionable operations may be displayed on the display of the electronic device, but is not limited thereto, and the list may also be output as sound by an audio output module of the electronic device. According to one embodiment, the order in which operations are listed in the list may be determined by the priority of the operations. For example, operations with higher priority may be located at the top of the list. According to one embodiment, an operation that has a history of being executed in the electronic device may have a higher priority than an operation that has no history of being executed. According to one embodiment, when a list containing operations that have no history of being executed is displayed in the electronic device and a user selects an operation from the list, information indicating that the operation has been selected may be reflected in a database, and the database may be referenced when operations are subsequently identified in the electronic device or listed in the list. The database can store information about actions, and the information about actions may include information about the priority of the actions. Actions previously selected by the user have a higher priority than actions that are not selected, and by referring to the database, actions with a higher priority can be identified or placed at the top of the list. Various scenarios for determining an activity to be performed by the user based on an image of the user's hand and objects around the hand captured by the electronic device, and for identifying and executing an action that is executable on the electronic device in response to the determined activity, will be described later with reference to FIGS. 2, FIG. 3, FIG. 5a, FIG. 5b, FIG. 6, and FIG. 7.

[0060] According to one embodiment, a method for determining an activity to be performed by a user using an object (204, 304, 503, 504, 605, 703, 704, 705) may be provided. The method may include the operation of acquiring an image (200, 300, 400, 500a, 500b, 600, 700) captured by an electronic device (801, 901) of the user's hand (602, 302, 501, 502, 601, 602, 701, 702) and an object (204, 304, 503, 504, 605, 703, 704) held by the hand. The above method may include an action of obtaining activity information (429) regarding at least one activity to be performed by the user using the object, based on the attributes of the object in the image and the form of the hand for grasping the object. The above method may include an action of identifying at least one application (946) that corresponds to the at least one activity and is executable on the electronic device, based on the activity information. The above method may include an action of executing an application included in the identified at least one application on the electronic device.

[0061] According to one embodiment, the operation of acquiring the activity information may include applying the image to a first artificial intelligence model (420, 422) to acquire interaction information regarding the shape of the object and the hand for grasping the object. The operation of acquiring the activity information may include applying the interaction information to a second artificial intelligence model (420, 424) to acquire the activity information regarding the at least one activity to be performed by the user using the object.

[0062] According to one embodiment, the operation of acquiring activity information may include an operation of acquiring environment information regarding the environment in which the user is located. The operation of acquiring activity information may include an operation of acquiring history information regarding the history of the user's use of the electronic device. The operation of acquiring activity information may include an operation of acquiring activity information regarding at least one activity to be performed by the user using the object by applying the interaction information, the environment information, and the history information to the second artificial intelligence model.

[0063] According to one embodiment, the electronic device may include an electronic device (801, 901) that can be worn on the user's head.

[0064] According to one embodiment, the electronic device includes a camera (880, 980) and a depth sensor (876, 976), and the image may include a photograph obtained through the camera and depth information obtained through the depth sensor.

[0065] According to one embodiment, the object may include a tool (204, 304, 503, 504, 605, 703, 704, 705) that is not communicationally connected to the electronic device.

[0066] According to one embodiment, the method may include an operation of changing a setting value of the electronic device corresponding to at least one activity.

[0067] According to one embodiment, the method may include an operation of playing multimedia content corresponding to at least one activity.

[0068] According to one embodiment, the method may include providing activity information regarding the at least one activity to a server (434, 908) based on the fact that the at least one application corresponding to the at least one activity is not identified in the electronic device. The method may include receiving information regarding the application corresponding to the at least one activity from the server.

[0069] According to one embodiment, the object is a first object, and the image may include a second object (705) that is not held in the user's hand. When the image is applied to an artificial intelligence model, a greater weight may be assigned to the first object than to the second object.

[0070] FIG. 2 is a diagram illustrating a method for determining an activity to be performed using a user's right hand with an object, according to one embodiment.

[0071] Referring to FIG. 2, an image (200) obtained from an electronic device (e.g., a wearable electronic device worn on the head or face) may include the user's right hand (202) and an object (204) around the right hand (202). The object (204) may be grasped by the right hand (202) as shown in FIG. 2.

[0072] Referring to FIG. 2, the user's right hand (202) may grasp a short rod (204) using three fingers, for example, the thumb, index finger, and middle finger, and a posture may be formed in which the thumb and index finger are adjacent to form a loop or hook shape.

[0073] According to one embodiment, the electronic device can analyze the acquired image (200) to determine that the type of object (204) is a short rod (204), and can determine that the posture of the right hand (202) for grasping the object (204) is a gripping method using three fingers, that is, a tripod grip. The electronic device can output the result of the determination as interaction information. For example, the artificial intelligence model of the electronic device (e.g., the first artificial intelligence model (422) of FIG. 4a) can analyze the image (200) to determine the attributes of the object and the shape of the hand for grasping the object, and can output the result of the determination as interaction information.

[0074] According to one embodiment, interaction information represents an interaction between an object and a user, and for example, the interaction information may include, but is not limited to, object information regarding the object, posture information regarding the posture of a hand for grasping the object, wearing information regarding the way the object is worn or attached to the hand, and motion information regarding the movement of the hand, and may include various information to describe the interaction between the object and the user.

[0075] According to one embodiment, the electronic device can determine at least one activity to be performed by a user using an object based on interaction information and can output activity information regarding the determined activity. For example, the artificial intelligence model of the electronic device (e.g., the activity inference module (424) of FIG. 4a) can infer at least one activity to be performed by a user using an object based on interaction information output from the first artificial intelligence model (422) and can output activity information regarding the inferred activity.

[0076] For each activity (e.g., all activities stored in the activity inference module (424)), the artificial intelligence model can calculate the probability that matches the short bar (204) and the three-finger method for grasping the short bar (204), and can infer that the activity with the highest calculated probability or the calculated probability exceeding a threshold is the activity to be performed by the user.

[0077] Referring to FIG. 2, based on the fact that the type of object (204) is a short rod (204) and the posture of the right hand (202) for gripping the object (204) is a three-finger method using three fingers, the activity to be performed by the user can be determined to be writing.

[0078] If the object (204) is a table tennis racket, even if the posture of the right hand (202) is the same as the three-finger method, the activity to be performed by the user can be determined to be table tennis. If the activity to be performed by the user is determined to be table tennis, the electronic device can identify and execute actions that correspond to table tennis and are executable on the electronic device. For example, the electronic device can run a table tennis game application or play table tennis-related content.

[0079] If it is determined that the activity to be performed by the user is handwriting, the electronic device may correspond to the handwriting and identify and execute actions that are executable on the electronic device. For example, the electronic device may be set to handwriting mode or drawing mode, or a note application or a drawing application may be launched.

[0080] According to one embodiment, the electronic device can determine an activity to be performed by the user using an object based on an image of the user's hand and an object around the hand, even in an executed application, and can identify and execute an action that can be performed by the electronic device in response to the determined activity. For example, in an electronic device in which a drawing application is executed or a user holds a short rod (204) with their right hand (202) as in FIG. 2 and is set to drawing mode, the user can change the posture of holding the rod (204), and the process of determining an activity to be performed by the user after changing the posture is described with reference to FIG. 3.

[0081] FIG. 3 is a diagram illustrating a method for determining an activity to be performed after a user changes their posture, according to one embodiment.

[0082] Referring to FIG. 3, compared to FIG. 2, the area where the rod (304) is exposed from the hand (302) is larger, and the position of the hand (302) is changed so that the index finger of the hand (302) is more extended compared to the hand (202) in the pencil grip position of FIG. 2, and the hand (302) can take an overhand grip position in which the rod (304) is wrapped with the entire finger.

[0083] As shown in FIG. 2, in an electronic device in which a user holds a short rod (204) with their right hand (202) and is set to drawing mode or a drawing application is executed, an image (300) obtained from the electronic device (e.g., a wearable electronic device worn on the head or face) after the user changes their posture may include the user's right hand (302) and objects (304) around the right hand (302), and may include the user's right hand (302) holding the short rod (304) with an overhand grip.

[0084] The electronic device can analyze the acquired image (300) to determine that the type of object (304) is still the same rod (304), and can determine that the posture of the right hand (302) for grasping the object (304) has changed to an overhand grip, and can output the result of the determination as interaction information. For example, the artificial intelligence model of the electronic device (e.g., the first artificial intelligence model (422) of FIG. 4a) can analyze the acquired image (300) to determine the attributes of the object (304) included in the image (300) and the shape of the hand for grasping the object (304), and can output the result of the determination as interaction information.

[0085] According to one embodiment, the electronic device can determine at least one activity to be performed by a user using an object based on interaction information and can output activity information regarding the determined activity. For example, the artificial intelligence model of the electronic device (e.g., the activity inference module (424) of FIG. 4a) can infer at least one activity to be performed by a user using an object based on interaction information output from the first artificial intelligence model (422) and can output activity information regarding the inferred activity.

[0086] For each activity (e.g., all activities stored in the activity inference module (424)), the artificial intelligence model can calculate the probability of matching the short rod (304) and the overhand grip for grasping the short rod (304), and can infer that the activity with the highest calculated probability or the calculated probability exceeding a threshold is the activity to be performed by the user.

[0087] Referring to FIG. 3, based on the fact that the type of object (304) is a short rod (304) and the position of the right hand (302) for holding the object (304) is an overhand grip, the activity to be performed by the user can be determined to be brushing.

[0088] If the determined activity is determined to be a brush stroke, the electronic device can correspond to the brush stroke and identify and execute an action that can be performed on the electronic device. For example, on an electronic device set to drawing mode or running a drawing application, the input pen can be changed from a handwriting pen to a brush.

[0089] FIG. 4a is a diagram illustrating a method for obtaining activity information regarding an activity to be performed by a user using an object, according to one embodiment.

[0090] According to one embodiment, the electronic device may include an image processing module (410) and an artificial intelligence model (420). The image processing module (410) and the artificial intelligence model (420) may be stored in the memory of the electronic device and driven by the processor of the electronic device.

[0091] According to one embodiment, an image processing module (410) can process an image (400) including a user's hand and the area around the hand so that it is suitable for use by an artificial intelligence model (420), and then transmit the processed image (400) to the artificial intelligence model (420). The image (400) may include an image obtained from a camera of an electronic device and depth information obtained from a depth sensor of an electronic device, and the image processing module (410) can transmit the image obtained from the camera and the depth information obtained from the depth sensor, or fuse the image obtained from the camera and the depth information obtained from the depth sensor and transmit the fused information to the artificial intelligence model (420).

[0092] According to one embodiment, the artificial intelligence model (420) can infer an activity to be performed by a user based on received information. The artificial intelligence model (420) can output activity information (429) regarding the activity as a result of the inference.

[0093] According to one embodiment, the artificial intelligence model (420) may include a first artificial intelligence model (422) for outputting interaction information regarding the posture of an object and a hand based on received information (e.g., an image (400)). The first artificial intelligence model (422) may be pre-trained to identify the posture of an object and a hand included in the information based on the received information.

[0094] According to one embodiment, interaction information represents an interaction between an object and a user, and for example, the interaction information may include, but is not limited to, object information regarding the object, posture information regarding the posture of a hand for grasping the object, wearing information regarding the way the object is worn or attached to the hand, and motion information regarding the movement of the hand, and may include various information to describe the interaction between the object and the user.

[0095] According to one embodiment, the first artificial intelligence model (422) may include an object identification module (4222) for identifying an object included in received information (e.g., image (400)) and a posture identification module (4224) for identifying the posture of a hand around an object. The object identification module (4222) may be trained to identify the type of object included in the received information based on the received information. The posture identification module (4224) may be trained to identify the posture of a hand included in the received information based on the received information. FIG. 4a illustrates that the object and the posture of the hand are identified by separate modules (4222 and 4224), but the object and the posture of the hand may be identified by a single module.

[0096] According to one embodiment, the object identification module (4222) and the posture identification module (4224) can each output object information regarding an object and posture information regarding the posture of a hand based on received information (e.g., image (400)). The posture information regarding the posture of the hand may include wearing information regarding the manner in which the object is worn or attached to the hand and motion information regarding the movement of the hand. According to one embodiment, the first artificial intelligence model (422) can output interaction information regarding an object and the posture of a hand for grasping the object based on received information (e.g., image (400)).

[0097] According to one embodiment, the artificial intelligence model (420) may include a first artificial intelligence model (422) comprising an object identification module (4222) and a pose identification module (4224). The artificial intelligence model (420) may include a second artificial intelligence model (424) referred to as an activity inference module (424).

[0098] Information output by the first artificial intelligence model (422) or the object identification module (4222) and the posture identification module (4224) can be transmitted to the activity inference module (424). For example, interaction information regarding an object and the posture of a hand for grasping the object can be output and transmitted to the activity inference module (424).

[0099] According to one embodiment, the activity inference module (424) can infer an activity to be performed by a user based on received information (e.g., interaction information). The activity inference module (424) can output activity information (429) regarding the activity as a result of the inference. The activity inference module (424) can be pre-trained to infer an activity to be performed by a user based on received information.

[0100] According to one embodiment, the electronic device may include an environment information acquisition module (426). The environment information acquisition module (426) may be stored in the memory of the electronic device and driven by the processor of the electronic device.

[0101] According to one embodiment, the environment information acquisition module (426) may acquire or collect environment information regarding the environment where the user is located from an electronic device or another electronic device connected to the electronic device. The environment information may include location information indicating the location of the environment where the user is located, and the location information may be acquired using a GPS (global positioning system) sensor included in the electronic device or another electronic device connected to the electronic device. The environment information may include noise information indicating the noise level of the environment where the user is located, and the noise information may be acquired using a noise sensor or microphone included in the electronic device or another electronic device connected to the electronic device. The environment information may include temperature information indicating the temperature of the environment where the user is located, and the temperature information may be acquired using a temperature sensor included in the electronic device or another electronic device connected to the electronic device.

[0102] According to one embodiment, environmental information obtained through the environmental information acquisition module (426) can be transmitted to the artificial intelligence model (420). For example, environmental information obtained through the environmental information acquisition module (426) can be transmitted to the activity inference module (424), and the activity inference module (424) can output activity information (429) regarding an activity to be performed by a user based on the interaction information received from the first artificial intelligence model (422) and the environmental information received from the environmental information acquisition module (426).

[0103] According to one embodiment, the electronic device may include a history information acquisition module (428). The history information acquisition module (428) may be stored in the memory of the electronic device and driven by the processor of the electronic device.

[0104] According to one embodiment, the history information acquisition module (428) may acquire or collect history information regarding the history of a user's use of an electronic device. The history information may include behavior data that tracks how a user behaves on a specific platform (application or webpage), such as content input, function usage records, search history, webpage visit history, activity history, exercise history, and viewing history.

[0105] According to one embodiment, the history information acquisition module (428) can acquire history information from various applications running on the electronic device. For example, the content of a memo entered by the user can be acquired by a memo application. For example, the web pages visited by the user and search history can be acquired by a web browsing application. For example, the content of a schedule registered by the user can be acquired by a schedule application. For example, the application used by the user on the electronic device, the usage time of the application, and the frequency of use can be recorded by a screen time application. According to one embodiment, the user's activity history or exercise history, such as walking, running, driving, and sports, can be measured or recorded by an exercise application, a map application, a navigation application, or a health application, but is not limited thereto, and can be measured or recorded by various applications.

[0106] According to one embodiment, historical information may be archived in an application that functions as a source of such historical information, but is not limited thereto, and may also be archived in a manager application that comprehensively stores various types of data, and the manager application may run in the background of the user's electronic device.

[0107] According to one embodiment, history information obtained through the history information acquisition module (428) can be transmitted to the artificial intelligence model (420). For example, history information obtained through the history information acquisition module (428) can be transmitted to the activity inference module (424), and the activity inference module (424) can output activity information (429) regarding an activity to be performed by a user based on the interaction information received from the first artificial intelligence model (422) and the history information received from the history information acquisition module (428).

[0108] According to one embodiment, the activity inference module (424) can output activity information (429) regarding an activity to be performed by a user based on interaction information received from the first artificial intelligence model (422), environment information received from the environment information acquisition module (426), and history information received from the history information acquisition module (428).

[0109] According to one embodiment, the electronic device may include a biometric information acquisition module (not shown) for acquiring or collecting a user's biometric information (e.g., heart rate information), and the biometric information acquisition module may acquire the user's biometric information measured by the electronic device or another wearable electronic device connected to the electronic device, and transmit the acquired biometric information to an artificial intelligence model (420).

[0110] According to one embodiment, the activity inference module (424) can output activity information (429) regarding an activity to be performed by a user based on interaction information received from the first artificial intelligence model (422), environment information received from the environment information acquisition module (426), history information received from the history information acquisition module (428), and biometric information acquired through the biometric information acquisition module.

[0111] According to one embodiment, the first artificial intelligence model (422) and the second artificial intelligence model (424) may both be stored in an electronic device and may be directly accessed by the electronic device, but are not limited thereto. For example, the first artificial intelligence model (422) may be stored in an electronic device and the second artificial intelligence model (424) may be stored in a server, the first artificial intelligence model (422) may be directly accessed by the electronic device, and the electronic device may receive information output by the second artificial intelligence model (424) from the server by querying the server containing the second artificial intelligence model (424). For example, the first artificial intelligence model (422) and the second artificial intelligence model (424) may both be stored in a server, and the electronic device may receive information output by the first artificial intelligence model (422) and / or the second artificial intelligence model (424) from the server by querying the server containing the first artificial intelligence model (422) and the second artificial intelligence model (424). The query transmitted from the electronic device to the server may include information input to the first artificial intelligence model (422) and / or the second artificial intelligence model (424), for example, an image (400), environmental information, history information, and / or biometric information.

[0112] FIG. 4b is a diagram illustrating a method for recommending an action to be performed in an electronic device based on inferred activity information according to one embodiment.

[0113] According to one embodiment, the electronic device may include a motion recommendation module (430) and a database (432), but is not limited thereto. For example, the electronic device may further include an event handler (440) and a framework (450). The motion recommendation module (430), the database (432), the event handler (440), and the framework (450) may be stored in the memory of the electronic device and may be driven or accessed by the processor of the electronic device.

[0114] According to one embodiment, the action recommendation module (430) receives activity information (429) output by the activity inference module (424) of FIG. 4a and can identify an action that corresponds to the activity indicated by the activity information (429) and is executable on the electronic device. To identify an action that corresponds to the activity indicated by the activity information (429) and is executable on the electronic device, the electronic device can query the database (432).

[0115] According to one embodiment, the database (432) may store information regarding activities performed by a user and operations of an electronic device corresponding to those activities. The information regarding activities performed by a user and operations of an electronic device corresponding to those activities may include information regarding activities actually performed by the user and operations executed by the electronic device in conjunction with those activities. The electronic device may identify the operations of the electronic device corresponding to the user's activities through the database (432), and thereby, operations that meet the user's needs may be executed.

[0116] According to one embodiment, if, as a result of querying the database (432), there is no action that can be executed on the electronic device corresponding to the activity indicated by the activity information (429), the electronic device may query the server (434). The server (434) may respond to the electronic device with an action that can be executed on the electronic device corresponding to the activity indicated by the activity information (429).

[0117] According to one embodiment, an action executable on an electronic device may be the installation and / or execution of an application corresponding to the activity indicated by the activity information (429), but is not limited thereto. For example, an action executable on an electronic device may include the playback of multimedia content corresponding to the activity indicated by the activity information (429), the change of a setting value of the electronic device, the transmission of a message, and a telephone connection. An action executable on an electronic device may refer to an action executable at the operating system level of the electronic device.

[0118] According to one embodiment, the action recommendation module (430) can recommend an action corresponding to an activity indicated by activity information (429) by querying the database (432) or by querying the database (432) and the server (434). Information regarding the recommended action can be transmitted to an event handler (440). The event handler (440) can pre-register a handler to be called when a specific event (e.g., receiving information regarding the recommended action) occurs in the electronic device, and when the event occurs, call the registered handler to perform a defined action. The event handler (440) can efficiently perform the defined action using a framework (450), and the framework (450) can maintain consistency in the execution of actions in the electronic device by effectively managing the event handler and system resources and providing the necessary interfaces.

[0119] FIG. 5a is a diagram illustrating a method for inferring an activity to be performed using two objects with both hands of a user, according to one embodiment.

[0120] Referring to FIG. 5a, an image (500a) obtained from an electronic device (e.g., a wearable electronic device worn on the head or face) includes the user's two hands (501 and 502) and objects (503 and 504) around them. The objects (503 and 504) can be grasped by the two hands (501 and 502) as shown in FIG. 5a.

[0121] Referring to FIG. 5a, a posture can be formed in which the user's two hands (501 and 502) grasp a long rod (503 and 504) using their entire fingers.

[0122] An artificial intelligence model of an electronic device (e.g., the first artificial intelligence model (422) of FIG. 4a) can analyze the received image (500a) to determine that the type of object (503 and 504) is a long rod (503 and 504), and can determine that the posture of both hands (501 and 502) for grasping the object (503 and 504) is a matched grip in which the backs of both hands are facing upward, the thumbs are placed on the upper part of the rod (503 or 504), and the index and middle fingers wrap around the lower part of the rod (503 or 504), and can output the result of the determination as interaction information.

[0123] According to one embodiment, an artificial intelligence model of an electronic device (e.g., an activity inference module (424) of FIG. 4a) can infer at least one activity to be performed by a user using an object based on interaction information output from a first artificial intelligence model (422)), and can output activity information regarding the inferred activity.

[0124] For each activity (e.g., all activities stored in the activity inference module (424)), the artificial intelligence model can calculate the probability of matching two long rods (503 and 504) and a matched grip for holding them, and can infer that the activity with the highest calculated probability or the calculated probability exceeding a threshold is the activity to be performed by the user.

[0125] Referring to FIG. 5a, based on the fact that the objects (503 and 504) are of the type two long rods (503 and 504) and the posture of both hands (501 and 502) for holding them is a matched grip for holding drumsticks, the activity to be performed by the user can be determined to be playing drums.

[0126] If the determined activity is determined to be drum playing, the electronic device may identify and execute actions that correspond to drum playing and are executable on the electronic device. For example, the electronic device may be set to instrument playing mode, a drum-related application (e.g., a drum playing application or a drum sheet music application) may be executed, or drum-related content may be played. According to one embodiment, the electronic device (e.g., the activity inference module (424) of FIG. 4a) may determine (e.g., infer) an activity to be performed by the user using an object by further considering environmental information in addition to interaction information. Environmental information may include information regarding whether the user is indoors or outdoors, or noise information. If the user is indoors or is presumed to be indoors due to low noise levels, the activity to be performed by the user may be determined (e.g., inferred) to be instrument playing. On the other hand, if the user is outdoors or is presumed to be outdoors due to high noise levels, the activity to be performed by the user may be determined (e.g., inferred) to be exercise. For example, the activity to be performed by the user using two rods (503 and 504) can be determined (e.g., inferred) to be jump rope. The electronic device can perform actions that correspond to jump rope and are executable on the electronic device, such as setting the electronic device to exercise mode, playing jump rope-related content on the electronic device, or running a jump rope application.

[0127] According to one embodiment, an activity to be performed by a user can be determined by considering environmental information, so that an operation that meets the user's needs can be executed on an electronic device.

[0128] According to one embodiment, when a plurality of executable actions are identified in an electronic device, an action having a higher priority (e.g., an action more associated with an activity determined to be performed by the user) may be executed, or a countdown may begin in order of priority, starting with the action having the highest priority. For example, if it is determined by environmental information that the user is indoors, a drum-related application (e.g., a drum playing application or a drum sheet music application) may have a higher priority than a jump rope application, and a countdown for the drum-related application may begin first. When the countdown ends without user intervention, the drum-related application may be executed.

[0129] According to one embodiment, a drum-related application is executed on an electronic device according to a determined activity, and in an image (500a) displayed for a user through an electronic device (e.g., HMD), an object (e.g., two long rods (503 and 504)) and / or its attributes (e.g., appearance) may be displayed with changes. For example, the changed appearance of the object (503 and 504) may be displayed by extracting feature points of the object (503 and 504), generating a 2D or 3D mapping based on the extracted feature points, and overlaying the changed appearance on the existing appearance based on the 2D or 3D mapping, but the changed appearance of the object (503 and 504) may be displayed using various other methods. According to one embodiment, the properties of the objects (503 and 504) may include, but are not limited to, the appearance, shape, size, color, texture, material, physical properties, density, weight, hardness, flexibility, elasticity, viscosity, and / or transparency of the objects (503 and 504). Changing and displaying the properties of the objects (503 and 504) is described with further reference to FIG. 5b.

[0130] FIG. 5b is a drawing for explaining a method of displaying an image according to an activity to be performed using two objects with both hands of a user, according to one embodiment.

[0131] Referring to FIG. 5b, an image (500b) displayed for a user through an electronic device (e.g., HMD) may be displayed in a see-through or pass-through manner, and the image (500b) may include actual objects (520, 522 and 524) visible through the see-through or pass-through.

[0132] The objects (503 and 504) corresponding to the two rods held in both hands in FIG. 5a can be displayed as two drumsticks. FIG. 5b illustrates that the appearance of the rods has been changed to wooden drumsticks with round tips, but is not limited thereto, and the objects can be displayed as drumsticks of various shapes. For example, the material of the drumsticks (e.g., wood, plastic, carbon fiber), length (e.g., a standard 16-inch stick, a short stick less than 16 inches, a long stick greater than 16 inches), thickness, and tip shape may vary. According to one embodiment, not only the appearance of the objects but also the properties of the objects may be changed. For example, depending on the material of the drumsticks, the sound produced by the drumsticks may also be output differently.

[0133] According to one embodiment, an image (500b) displayed in a see-through or pass-through manner may display a virtual object (510) based on an activity to be performed by a user using the object. For example, referring to FIG. 5b, if it is determined that an activity to be performed by a user using two long sticks is drum playing, a drum-related application may be executed, and a virtual object (510) for drum playing, for example, a virtual drum (510), may be represented (e.g., rendered) in the image (500b). Thus, the user may play the virtual drum (510) using drumsticks corresponding to the two long sticks.

[0134] According to one embodiment, a user interface for changing the properties (e.g., appearance, shape, size, color, texture, material, physical properties, density, weight, hardness, flexibility, elasticity, viscosity, and / or transparency) of an object held by the user's hand, a real object (520, 522 and 524), and / or a virtual object (510) may be provided to the user. For example, a drum-related application running on an electronic device may provide a user interface for changing or purchasing the properties of a drum stick and / or a virtual drum (510).

[0135] FIG. 6 is a diagram illustrating a method for determining an activity to be performed using both hands of a user with a single object, according to one embodiment.

[0136] Referring to FIG. 6, an image (600) obtained from an electronic device (e.g., a wearable electronic device worn on the head or face) includes the user's two hands (601 and 602) and an object (605) around them. The object (605) can be grasped by both hands (601 and 602) as shown in FIG. 6.

[0137] Referring to FIG. 6, a posture can be formed in which the user's two hands (601 and 602) grasp a long rod (605) using their fingertips.

[0138] The electronic device can analyze the acquired image (600) to determine that the type of object (605) is a long rod (605), and can determine that the posture of both hands (601 and 602) for grasping the object (605) is a flute-playing posture with fingertips touching a single long rod (605), and can output the result of the determination as interaction information. For example, the artificial intelligence model of the electronic device (e.g., the first artificial intelligence model (422) of FIG. 4a) can analyze the acquired image (600) to determine the attributes of the object (605) included in the image (600) and the shape of the hands for grasping the object (605), and can output the result of the determination as interaction information.

[0139] According to one embodiment, the electronic device can determine at least one activity to be performed by a user using an object based on interaction information and can output activity information regarding the determined activity. For example, the artificial intelligence model of the electronic device (e.g., the activity inference module (424) of FIG. 4a) can infer at least one activity to be performed by a user using an object based on interaction information output from the first artificial intelligence model (422)) and can output activity information regarding the inferred activity.

[0140] For each activity (e.g., all activities stored in the activity inference module (424)), the artificial intelligence model can calculate the probability that matches the long rod (605) and the flute-playing posture for holding it, and can infer that the activity with the highest calculated probability or the calculated probability exceeding a threshold is the activity to be performed by the user.

[0141] Referring to FIG. 6, based on the fact that the type of object (605) is a long rod (605) and the posture of both hands (601 and 602) for holding it is a flute-playing posture for holding a flute, the activity to be performed by the user can be determined to be playing the flute.

[0142] If the determined activity is determined to be playing the flute, the electronic device can correspond to playing the flute and identify and execute actions that are executable on the electronic device. For example, the electronic device can be set to instrument playing mode, a flute sheet music application can be launched, or flute-related content can be played.

[0143] According to one embodiment, an electronic device (e.g., the activity inference module (424) of FIG. 4a) can determine (e.g., infer) an activity to be performed by a user using an object by further considering history information in addition to interaction information. History information may include behavior data that tracks how a user behaves on a specific platform (application or webpage), such as content input, function usage history, search history, webpage visit history, activity history, exercise history, and viewing history. For example, if a user has previously run a flute sheet music application while playing a flute, turned off notifications to focus on playing the flute, run a metronome application in the background, or played accompaniment content, the activity to be performed by the user is playing a musical instrument, and among them, it can be determined (e.g., inferred) that it is playing a flute.

[0144] According to one embodiment, an activity to be performed by a user can be determined by considering historical information, so that an operation that meets the user's needs can be executed on an electronic device.

[0145] According to one embodiment, when a plurality of actions executable on an electronic device are identified, the identified actions may be executed sequentially or in parallel on the electronic device. The actions executed sequentially or in parallel may be mutually compatible actions. For example, an action that can be run in the background or in picture-in-picture (PIP) mode may be executed together with other actions. Accordingly, the electronic device may dismiss a notification, launch a flute sheet music application, play accompaniment content, and run a metronome in the background or in PIP mode.

[0146] FIG. 7 is a diagram illustrating a method for determining an activity to be performed by a user using two or more objects according to one embodiment.

[0147] The electronic device can analyze the acquired image (700) to determine that the types of objects (703, 704, and 705) are food ingredients (703) and cooking tools (704 and 705), and can output the result of the determination as interaction information. For example, the artificial intelligence model of the electronic device (e.g., the first artificial intelligence model (422) of FIG. 4a) can analyze the acquired image (700) to determine the attributes of the objects (703, 704, and 705) and the shape of the hand for grasping the objects (703 and 704), and can output the result of the determination as interaction information. The interaction information may include information about objects (705) located around the hand but not grasped by the hand.

[0148] According to one embodiment, the electronic device can determine at least one activity to be performed by a user using an object based on interaction information and can output activity information regarding the determined activity. For example, the artificial intelligence model of the electronic device (e.g., the activity inference module (424) of FIG. 4a) can infer at least one activity to be performed by a user using an object based on interaction information output from the first artificial intelligence model (422)) and can output activity information regarding the inferred activity.

[0149] For each activity (e.g., all activities stored in the activity inference module (424)), the artificial intelligence model can calculate the probability of matching with the ingredients (703) and cooking tools (704 and 705), and can infer that the activity with the highest calculated probability or the calculated probability exceeding a threshold is the activity to be performed by the user.

[0150] Referring to FIG. 7, the types of objects (705) are food ingredients (703) and cooking tools (704 and 705), and based on the form in which both hands (701 and 702) hold the food ingredients (703) or cooking tools (704) and the position of the cooking tools (705) around both hands (701 and 702), the activity to be performed by the user can be determined to be cooking. According to one embodiment, the artificial intelligence model may assign a greater weight to the object (703 or 704) held by the hand (701 or 702) than to the object (705) not held by the hand (701 or 702). FIG. 7 shows that both the grasped object (703 or 704) and the ungrasped object (705) are related to cooking, so the activity to be performed by the user is inferred to be cooking; however, if the activity corresponding to the grasped object (703 or 704) and the activity corresponding to the ungrasped object (705) are different, the activity corresponding to the grasped object (703 or 704), to which a higher weight is assigned, can be inferred to be the activity to be performed by the user. According to one embodiment, different weights may be assigned to the objects depending on the distance between the user's hand and the objects. For example, the closer an object is to the user's hand, the greater the weight may be assigned. For example, if the distance between an object (703 or 704) held by the user's hand and the user's hand is 0 cm, and the distance between an object (705) not held by the user's hand and the hand is 3 cm, the weight assigned to the object (705) further away from the user's hand may have a smaller value than the weight assigned to the object (703 or 704) closer to the user's hand.

[0151] If it is determined that the activity to be performed by the user is cooking, the electronic device may identify and execute actions that correspond to cooking and are executable on the electronic device. For example, the electronic device may be set to cooking mode, a cooking information application may be launched, cooking-related content may be played, or a timer application may be launched when the user's hand is near a pot or burner.

[0152] According to one embodiment, based on an image including an object and a hand wearing the object, an activity to be performed by a user using the object can be inferred. For example, based on an image of a user wearing boxing gloves on their hand, the activity to be performed by the user can be inferred to be boxing, and actions corresponding to boxing and executable on an electronic device, such as the execution of a health application, a timer application, or an exercise log application, or the playback of boxing-related content, can be executed.

[0153] According to one embodiment, an artificial intelligence model may assign a greater weight to an object held by a hand than to an object not held by a hand. For example, if an object held by a hand is a drumstick and an object not held by a hand is a piano, a greater weight is assigned to the drumstick, and the artificial intelligence model may infer that the activity to be performed by the user is playing the drums rather than playing the piano.

[0154] FIG. 8 is a block diagram of a wearable electronic device according to one embodiment.

[0155] An electronic device according to one embodiment may be the wearable electronic device (801) of FIG. 8. Referring to FIG. 8, the wearable electronic device (801) may include a processor (820), memory (830), a sensor module (876), a camera module (880), and a communication module (890). The wearable electronic device (801) may include additional components in addition to the components shown in FIG. 8. Some of the components of the wearable electronic device (801) shown in FIG. 8 may be omitted, replaced, or integrated with one another. The wearable electronic device (801) may correspond to the electronic device (901) of FIG. 9.

[0156] According to one embodiment, the components of the wearable electronic device (801) shown in FIG. 8 may correspond to the components of the electronic device (901) shown in FIG. 9. For example, the processor (820), memory (830), sensor module (876), camera module (880), and communication module (890) may correspond to the processor (920), memory (930), sensor module (976), camera module (980), and communication module (990) of FIG. 9, respectively.

[0157] According to one embodiment, components included in a wearable electronic device (801) may be electrically and / or operatively connected to each other to exchange signals (e.g., commands or data) with one another. For example, instructions stored in memory (830) may be executed by a processor (820). For example, the processor (820) may execute instructions stored in memory (830) to perform a specified function (or logic) or control other components of the wearable electronic device (801).

[0158] In one embodiment, the sensor module (876) may be a depth sensor, and depth information may be obtained by photographing the user's hand and objects around the hand through the depth sensor. The depth sensor may be an infrared sensor, a Time-of-Flight (ToF) sensor, an ultrasonic sensor, a light detection and ranging (LiDAR) sensor, or a stereo camera, and depth information may be generated by the depth sensor. The sensor module (876) may transmit the obtained depth information to the processor (820).

[0159] In one embodiment, the camera module (880) can capture a subject (e.g., a hand and objects around the hand) to acquire an image (e.g., a still image or a video). The camera module (880) can transmit the acquired image to the processor (820).

[0160] In one embodiment, the communication module (890) can be used for the wearable electronic device (801) to communicate with another electronic device or server.

[0161] According to one embodiment, an electronic device (or wearable electronic device) (801, 901) may be provided, comprising one or more processors (820, 920) including a memory (830, 930) for storing instructions and processing circuitry. When the above commands are executed individually or collectively by the one or more processors, the electronic device may be enabled to acquire, through the electronic device, an image (200, 300, 400, 500a, 500b, 600, 700) of a user's hand (602, 302, 501, 502, 601, 602, 701, 702) and an object (204, 304, 503, 504, 605, 703, 704) that has been captured by the hand. When the above commands are executed by the one or more processors, the electronic device may be enabled to acquire activity information (429) regarding at least one activity to be performed by the user using the object, based on the attributes of the object in the image and the form of the hand for grasping the object. When the above instructions are executed by the one or more processors, the electronic device may identify, based on the activity information, at least one application (946) that corresponds to the at least one activity and is executable on the electronic device. When the above instructions are executed by the one or more processors, the electronic device may execute an application included in the identified at least one application on the electronic device.

[0162] According to one embodiment, when the instructions are executed individually or jointly by the one or more processors, the electronic device may be enabled to: apply the image to a first artificial intelligence model (420, 422) to obtain interaction information regarding the attributes of the object and the shape of the hand for grasping the object. When the instructions are executed individually or jointly by the one or more processors, the electronic device may be enabled to apply the interaction information to a second artificial intelligence model (420, 424) to obtain activity information regarding at least one activity to be performed by the user using the object.

[0163] According to one embodiment, when the instructions are executed individually or jointly by the one or more processors, the electronic device may be enabled to: acquire environmental information regarding the environment in which the user is located. When the instructions are executed individually or jointly by the one or more processors, the electronic device may be enabled to acquire history information regarding the history of the user's use of the electronic device. When the instructions are executed individually or collectively by the one or more processors, the electronic device may be enabled to apply the interaction information, the environmental information, and the history information to the second artificial intelligence model to acquire activity information regarding the at least one activity to be performed by the user using the object.

[0164] According to one embodiment, the electronic device may be wearable on the user's head.

[0165] According to one embodiment, the electronic device includes a camera (880, 980) and a depth sensor (876, 976), and the image may include a photograph obtained through the camera and depth information obtained through the depth sensor.

[0166] In the electronic device according to one embodiment, the object may include a tool (204, 304, 503, 504, 605, 703, 704, 705) that is not communicationally connected to the electronic device.

[0167] According to one embodiment, when the instructions are executed individually or jointly by the one or more processors, the electronic device may be able to change a setting value of the electronic device corresponding to the at least one activity.

[0168] According to one embodiment, when the instructions are executed individually or jointly by the one or more processors, the electronic device may be able to play multimedia content corresponding to the at least one activity.

[0169] According to one embodiment, when the instructions are executed individually or collectively by the one or more processors, the electronic device may provide the activity information regarding the at least one activity to the server (434, 908) based on the fact that the at least one application corresponding to the at least one activity is not identified in the electronic device. When the instructions are executed individually or collectively by the one or more processors, the electronic device may receive information regarding the application corresponding to the at least one activity from the server.

[0170] In the electronic device according to one embodiment, the object is a first object, and the image may include a second object (705) that is not held in the user's hand. When the image is applied to an artificial intelligence model, a greater weight may be assigned to the first object than to the second object.

[0171] FIG. 9 is a block diagram of an electronic device (901) in a network environment (900) according to various embodiments. Referring to FIG. 9, in the network environment (900), the electronic device (901) may communicate with an electronic device (902) through a first network (998) (e.g., a short-range wireless communication network) or with an electronic device (904) or a server (908) through a second network (999) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (901) may communicate with the electronic device (904) through a server (908). According to one embodiment, the electronic device (901) may include a processor (920), memory (930), input module (950), sound output module (955), display module (960), audio module (970), sensor module (976), interface (977), connection terminal (978), haptic module (979), camera module (980), power management module (988), battery (989), communication module (990), subscriber identification module (996), or antenna module (997). In some embodiments, at least one of these components (e.g., connection terminal (978)) may be omitted from the electronic device (901), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (976), camera module (980), or antenna module (997)) may be integrated into a single component (e.g., display module (960)).

[0172] The processor (920) can control at least one other component (e.g., a hardware or software component) of the electronic device (901) connected to the processor (920) by executing software (e.g., a program (940)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (920) can store commands or data received from other components (e.g., a sensor module (976) or a communication module (990)) in volatile memory (932), process the commands or data stored in volatile memory (932), and store the resulting data in non-volatile memory (934). According to one embodiment, the processor (920) may include a main processor (921) (e.g., a central processing unit or an application processor) or an auxiliary processor (923) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (901) includes a main processor (921) and an auxiliary processor (923), the auxiliary processor (923) may be configured to use lower power than the main processor (921) or to be specialized for a designated function. The auxiliary processor (923) may be implemented separately from the main processor (921) or as part thereof.

[0173] The auxiliary processor (923) may control at least some of the functions or states associated with at least one component of the electronic device (901) (e.g., display module (960), sensor module (976), or communication module (990)) on behalf of the main processor (921) while the main processor (921) is in an inactive (e.g., sleep) state, or together with the main processor (921) while the main processor (921) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (923) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (980) or communication module (990)). According to one embodiment, the auxiliary processor (923) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (901) itself where the artificial intelligence is performed, or through a separate server (e.g., server (908)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0174] The number of processors (920) may be one or more. For example, the processor (920) may have the structure of a multi-core processor such as a dual core, quad core, or hexa core.

[0175] The processor (920) can control the operations of the electronic device (901) by executing instructions stored in memory (930). For example, the processor (920) may correspond to a plurality of processors that divide and collectively perform a plurality of operations among the processors.

[0176] The memory (930) can store various data used by at least one component of the electronic device (901) (e.g., processor (920) or sensor module (976)). The data may include, for example, software (e.g., program (940)) and input or output data for related commands. The memory (930) may include volatile memory (932) or non-volatile memory (934).

[0177] The program (940) may be stored as software in memory (930) and may include, for example, an operating system (942), middleware (944), or an application (946).

[0178] The input module (950) can receive commands or data to be used for a component of the electronic device (901) (e.g., processor (920)) from outside the electronic device (901) (e.g., user). The input module (950) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0179] The sound output module (955) can output a sound signal to the outside of the electronic device (901). The sound output module (955) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0180] The display module (960) can visually provide information to an external (e.g., user) of the electronic device (901). The display module (960) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (960) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0181] The audio module (970) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (970) can acquire sound through the input module (950) or output sound through the sound output module (955) or an external electronic device (e.g., electronic device (902)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (901).

[0182] The sensor module (976) can detect the operating state of the electronic device (901) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (976) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0183] The interface (977) may support one or more specified protocols that can be used for the electronic device (901) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (902)). According to one embodiment, the interface (977) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0184] The connection terminal (978) may include a connector through which the electronic device (901) can be physically connected to an external electronic device (e.g., electronic device (902)). According to one embodiment, the connection terminal (978) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0185] The haptic module (979) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (979) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0186] The camera module (980) can capture still images and video. According to one embodiment, the camera module (980) may include one or more lenses, image sensors, image signal processors, or flashes.

[0187] The power management module (988) can manage the power supplied to the electronic device (901). According to one embodiment, the power management module (988) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0188] The battery (989) can supply power to at least one component of the electronic device (901). According to one embodiment, the battery (989) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0189] The communication module (990) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (901) and an external electronic device (e.g., electronic device (902), electronic device (904), or server (908)), and the performance of communication through the established communication channel. The communication module (990) may include one or more communication processors that operate independently of the processor (920) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (990) may include a wireless communication module (992) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (994) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (904) through a first network (998) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (999) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (992) can identify or authenticate the electronic device (901) within a communication network such as the first network (998) or the second network (999) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (996).

[0190] The wireless communication module (992) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (992) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (992) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (992) can support various requirements specified in the electronic device (901), external electronic device (e.g., electronic device (904)), or network system (e.g., second network (999)). According to one embodiment, the wireless communication module (992) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0191] An antenna module (997) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (997) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (997) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (998) or a second network (999), may be selected from the plurality of antennas, for example, by a communication module (990). A signal or power may be transmitted or received between the communication module (990) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (997).

[0192] According to various embodiments, the antenna module (997) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0193] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0194] According to one embodiment, commands or data may be transmitted or received between the electronic device (901) and an external electronic device (904) through a server (908) connected to a second network (999). Each of the external electronic devices (902, or 904) may be the same or a different type of device as the electronic device (901). According to one embodiment, all or part of the operations performed on the electronic device (901) may be performed on one or more of the external electronic devices (902, 904, or 908). For example, if the electronic device (901) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (901) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (901). The electronic device (901) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (901) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (904) may include an Internet of Things (IoT) device. The server (908) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (904) or the server (908) may be included within a second network (999).The electronic device (901) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0195] An electronic device (901) according to one embodiment may correspond to an electronic device of FIGS. 1 to 8 (e.g., an electronic device (801) of FIG. 8), and the electronic device (901) may perform the operations of an electronic device of FIGS. 1 to 8 (e.g., an electronic device (801) of FIG. 8).

[0196] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains.

[0197] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs.

[0198] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0199] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0200] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0201] Various embodiments of the present document may be implemented as software (e.g., program (940)) comprising one or more instructions stored in a storage medium (e.g., internal memory (936) or external memory (938)) readable by a machine (e.g., electronic device (901)). For example, a processor (e.g., processor (920)) of the machine (e.g., electronic device (901)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0202] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0203] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices: Memory for storing instructions; and It includes one or more processors including processing circuitry, and When the above instructions are executed individually or collectively by the one or more processors, the electronic device: An image of the user's hand and an object held by the hand is obtained through the electronic device, and Based on the properties of the object in the above image and the form of the hand for grasping the object, activity information regarding at least one activity to be performed by the user using the object is obtained, and Based on the above activity information, identify at least one application that corresponds to the at least one activity and is executable on the electronic device, and An electronic device that enables an application included in at least one of the identified applications to be executed on the electronic device.

2. In Paragraph 1, When the above instructions are executed individually or jointly by the one or more processors, the electronic device: By applying the above image to a first artificial intelligence model, interaction information regarding the attributes of the object and the shape of the hand for grasping the object is obtained, and An electronic device that applies the above interaction information to a second artificial intelligence model to obtain the above activity information regarding the at least one activity to be performed by the user using the above object.

3. In Paragraph 2, When the above instructions are executed individually or jointly by the one or more processors, the electronic device: Obtaining environmental information regarding the environment where the above user is located, and The above user obtains history information regarding the history of using the electronic device, and An electronic device that applies the interaction information, the environment information, and the history information to the second artificial intelligence model to obtain the activity information regarding the at least one activity to be performed by the user using the object.

4. In Paragraph 1, The above electronic device is an electronic device comprising an electronic device wearable on the head of the user.

5. In Paragraph 1, The above electronic device includes a camera and a depth sensor, and The above image is an electronic device comprising a photograph acquired through the camera and depth information acquired through the depth sensor.

6. In Paragraph 1, The above object is an electronic device comprising a tool that is not communicationally connected to the electronic device.

7. In Paragraph 1, When the above instructions are executed individually or collectively by the one or more processors, the electronic device: An electronic device that changes a setting value of the electronic device corresponding to at least one activity based on the above activity information.

8. In Paragraph 1, When the above instructions are executed individually or jointly by the one or more processors, the electronic device: An electronic device that performs playback of multimedia content corresponding to at least one of the above activities.

9. In Paragraph 1, When the above instructions are executed individually or jointly by the one or more processors, the electronic device: Based on the fact that the at least one application corresponding to the at least one activity is not identified in the electronic device, the activity information regarding the at least one activity is provided to the server, and An electronic device that receives information regarding an application corresponding to at least one activity from the above server.

10. In Paragraph 1, The above object is the first object, and The above image includes a second object not held by the user's hand, and An electronic device in which, when the above image is applied to an artificial intelligence model, a greater weight is assigned to the first object than to the second object.

11. In Paragraph 10, An electronic device in which a first weight and a second weight assigned to the first object and the second object, respectively, are determined based on a first distance between the first object and the user's hand and a second distance between the second object and the user's hand.

12. As a method: The operation of acquiring an image of a user's hand and an object held by said hand through an electronic device; An action of obtaining activity information regarding at least one activity to be performed by the user using the object, based on the attributes of the object and the form of the hand for grasping the object in the above image; An action of identifying at least one application executable on the electronic device corresponding to at least one activity based on the above activity information; and A method comprising the operation of executing an application included in at least one identified application on the electronic device.

13. In Paragraph 1, The action of acquiring the above activity information is: The operation of applying the above image to a first artificial intelligence model to obtain interaction information regarding the attributes of the object and the shape of the hand for grasping the object; and A method comprising applying the above interaction information to a second artificial intelligence model to obtain the above activity information regarding the at least one activity to be performed by the user using the object.

14. In Paragraph 1, A method comprising a tool that is not communicationally connected to the electronic device.

15. An action of acquiring an image of the user's hand and an object held by the hand through an electronic device; An action of obtaining activity information regarding at least one activity to be performed by the user using the object, based on the properties of the object and the form of the hand for grasping the object in the above image; An action that corresponds to at least one activity and identifies at least one application executable on the electronic device; and A computer-readable recording medium having a program for executing a method comprising: executing an application included in at least one identified application on the electronic device.

Images