Electronic device for providing ai assistant, and operating method thereof

Abstract

The present disclosure provides an electronic device. The electronic device can: use a sensor so as to detect a first touch input in a first area of a display; display, in response to identifying that the first touch input continues for a first threshold time or longer and that the shape of the first touch input matches a user registration button, a visual signal indicating that the user registration button is being recognized in an area in which the first touch input is recognized; and execute, in response to identifying that the first touch input continues for a second threshold time or longer, an AI assistant set in the user registration button.

Description

Electronic device providing an AI assistant and its method of operation

[0001] The present disclosure relates to an electronic device that provides an AI assistant and a method of operating the same.

[0002] An AI assistant is software or a system based on artificial intelligence technology that understands user needs and performs appropriate responses or tasks. For example, an AI assistant can utilize Natural Language Processing (NLP) and Large-Scale Language Modeling (LLM) to analyze user questions and understand context, thereby providing answers in natural language. By learning from user data, an AI assistant can provide personalized services. For instance, an AI assistant can learn the functions frequently used by users, manage schedules, and provide various recommendations based on user preferences.

[0003] AI assistants are connected to the cloud to process large-scale data in real time and perform advanced computational tasks. Furthermore, by leveraging API technology on edge devices, AI assistants can integrate with various platforms (e.g., smart home devices, automobiles, mobile applications) to provide features that make users' daily lives more convenient.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0005] According to one embodiment of the present disclosure, an electronic device (101) comprises: a display; a sensor; a memory including at least one storage medium for storing instructions; and at least one processor including a processing circuit; wherein, when the instruction is executed individually or collectively by the at least one processor, the electronic device may: detect a first touch input in a first area of ​​the display using the sensor, and in response to identifying that the first touch input persists for a first threshold time and that the shape of the first touch input matches a user registration button, display a visual signal indicating that the user registration button is being recognized in the area where the first touch input is recognized, and in response to the first touch input persisting for a second threshold time or longer, cause the electronic device to execute an AI assistant set on the user registration button.

[0006] A method of an electronic device according to one embodiment of the present disclosure may include: detecting a first touch input in a first area of ​​a display using a sensor; displaying a visual signal indicating that the user button is being recognized in an area where the first touch input is recognized, in response to identifying that the first touch input persists for a first threshold time or longer and that the shape of the first touch input matches a user registration button; and executing an AI assistant set on the user registration button in response to the first touch input persisting for a second threshold time or longer.

[0007] According to one embodiment, a storage medium may be provided for storing at least one instruction readable by a computer. The at least one instruction may cause the electronic device to perform at least one operation when executed by at least a part of at least one processor of the electronic device. The at least one operation may include: detecting a first touch input in a first area of ​​a display using a sensor; displaying a visual signal representing a first application associated with a user registration button in the area where the first touch input is recognized in response to identifying that the first touch input persists for a first threshold time or longer and that the shape of the first touch input matches a user registration button; and displaying an execution screen of the first application in at least a part of the display in response to the first touch input persisting for a second threshold time or longer.

[0008] However, the problems to be solved in this disclosure are not limited to those mentioned above, and may be determined in various ways without departing from the scope and subject matter of this disclosure.

[0009] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0010] FIG. 2 is an example of a screen for calling an AI assistant using a user registration button in an electronic device according to one embodiment of the present disclosure.

[0011] FIG. 3 is a flowchart illustrating an AI assistant call operation of an electronic device according to one embodiment of the present disclosure.

[0012] FIG. 4 is an example screen of an electronic device setting a user registration button according to one embodiment of the present disclosure.

[0013] FIG. 5 is an example of a user feedback screen for a user registration button of an electronic device according to one embodiment of the present disclosure.

[0014] FIG. 6 is an example of a user registration button by another finger according to an embodiment of the present disclosure.

[0015] FIG. 7 is a flowchart illustrating the search operation of an AI assistant of an electronic device according to one embodiment of the present disclosure.

[0016] FIG. 8 is an example of an AI assistant search function screen using a user registration button of an electronic device according to one embodiment of the present disclosure.

[0017] FIG. 9 is an example of a multi-search function screen of an AI assistant using a user registration button of an electronic device according to one embodiment of the present disclosure.

[0018] FIG. 10 is an example of a search result screen of an AI assistant in an electronic device according to one embodiment of the present disclosure.

[0019] FIGS. 11a, 11b, and 11c are examples of use of a user registration button of an electronic device according to one embodiment of the present disclosure.

[0020] FIG. 12 is an example of sharing the execution result of an AI assistant of an electronic device with another electronic device according to one embodiment of the present disclosure.

[0021] In the following description, the attached drawings are referenced, and specific examples of implementation are illustrated within the drawings. Additionally, other examples may be used and structural modifications may be made without departing from the scope of the various examples.

[0022] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit the technical features of this disclosure. For example, a component expressed in the singular form should be understood as a concept including singular or plural components unless the context clearly indicates only the singular form.

[0023] In the present disclosure, each of the phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B, or C” may include any one of the items listed with the corresponding phrase, or all possible combinations thereof. The term “and / or” as used in the present disclosure should be understood to encompass any possible combination by one or more of the plurality of items listed with the corresponding term. Terms such as “first,” “second,” “first,” or “second” as used in the present disclosure may be used merely to distinguish a component from another component and do not limit the components in any other aspect (e.g., importance or order).

[0024] Where it is stated that any (e.g., 1st) component is “coupled,” “connected,” “linked,” “coupled,” “supported,” “connected,” or “contacted” with or without the terms “functionally” or “communicationly,” it includes not only cases where the component is directly coupled, connected, linked, coupled, supported, or contacted with the other component, but also cases where it is indirectly coupled, connected, linked, coupled, supported, or contacted through a third component.

[0025] Terms such as "include" or "have" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in this disclosure, and do not preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof. When a component is described as being located "on" another component, this includes not only cases where the component is in contact with the other component, but also cases where another component exists between the two components.

[0026] The expression “configured to” as used in this disclosure may be appropriately substituted, depending on the context, for example, “suitable for,” “capable of,” “designed to,” “modified to,” “made to,” or “capable of.” The term “configured to” does not necessarily mean only that which is “specially designed” in hardware. Instead, in some situations, the expression “device configured to” may mean that the device is “capable of” together with other devices or components. For example, the phrase “device configured (or set) to perform A, B, and C” may mean a device dedicated to performing the said operation, or a general-purpose device capable of performing various operations including said operation.

[0027] Terms used in this disclosure, such as "upper side," "lower side," and "front-rear direction," are defined based on the drawings, and the shape and position of each component are not limited by these terms.

[0028] Although the description in this disclosure is centered on specific embodiments, this disclosure is not limited to such specific embodiments and should be understood to encompass all various modifications, equivalents, and / or substitutions of the various embodiments described in this disclosure. In connection with the description of the drawings, similar reference numerals may be used for similar or related components.

[0029] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0030] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0031] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0032] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0033] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0034] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0035] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0036] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0037] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0038] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0039] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0040] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multi-media interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0041] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0042] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0043] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0044] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0045] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0046] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0047] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or largescale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0048] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0049] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0050] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0051] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0052] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0053] FIG. 2 is an example of a screen for calling an AI assistant using a user registration button in an electronic device according to one embodiment of the present disclosure.

[0054] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1) can call an AI assistant function immediately by inputting a user registration button while using the electronic device. The user registration button represents the hand shape of a user registered by user input. When the electronic device (101) detects the user's hand shape on the display screen, it determines that the virtual user registration button has been pressed and can immediately launch the AI ​​assistant.

[0055] At the first time point (T1), the electronic device (101) can detect a touch input to a first area of ​​the current screen (210) through the display.

[0056] At a second time point (T2), the electronic device (101) may display a temporal signal (e.g., a visual cue) indicating that it recognizes the user registration button in the first area where the touch input is recognized. The electronic device (101) may identify that it is recognizing the user registration button if the touch input matches a user registration button that has been previously set and registered by the user and is maintained for a first threshold time (e.g., T2-T1).

[0057] In one embodiment, the user registration button itself is not displayed on the display but can be detected at any location on the display, and the contact surface of the user touch input may represent a long press touch input that matches a hand shape previously registered by the user. In one embodiment, the contact surface of the user touch input detected by the capacitive sensor may be composed of one or more closed curves. The closed curves may be composed of values ​​having a certain capacitance or values ​​greater than a certain capacitance. For example, the user registration button may be defined by one or more shapes (lines) that constitute an area corresponding to a certain touch intensity.

[0058] In one embodiment, the user registration button may be defined based on the shape of a touch input entered during the user's setting registration process. The shape of the touch input may be an area (e.g., finger contact shape) that is contacted by the user's body (e.g., finger) on the display. The shape of the touch input may gradually increase in contact area on the display depending on the pressure applied to the touch, that is, the harder it is pressed. The user registration button may be defined as a set of one or more shapes (e.g., closed curves) recognized while the touch input is being pressed. Referring to FIG. 2, as an example of a touch input, multiple contact areas may be recognized by the touch input from a first time point (T1) when the touch input is detected. The electronic device (101) may determine whether the touch input matches the user registration button by comparing the shape of the contact area caused by the touch input with one or more shapes defined in the user registration button.

[0059] In one embodiment, the user registration button may be recognized regardless of the touch input location on the display. To clearly distinguish user intent, if the user registration button persists for a threshold time or longer, it may be determined that the action of pressing the user registration button has been performed. The user registration button may be set, changed, or deleted by a user setting action.

[0060] In one embodiment, the first threshold time may be set based on the result of learning the user's usage pattern. For example, the first threshold time may be 0.01 to 0.5 seconds.

[0061] In one embodiment, the visual signal may be circular. Depending on the embodiment, the shape, size, color, clarity, or movement of the visual signal may vary.

[0062] In one embodiment, the electronic device (101) may identify a touch input as a touch gesture for a first area in response to the touch input ending within a first threshold time. In one embodiment, the electronic device (101) may identify a touch input as a long press gesture in response to the determination that the touch input does not match a user registration button.

[0063] At the third time point (T3), the electronic device (101) may display a visual display that gradually increases in intensity over time in response to the continued detection of touch input. For example, the electronic device (101) may display the visual display gradually larger or clearer.

[0064] At the fourth time point (T4), the electronic device (101) can execute the AI ​​assistant set on the user registration button in response to the touch input being maintained for a second threshold time (e.g., T4-T1). Although the operation of the electronic device (101) has been described separately according to the flow of time, the user input, which is the action of clicking the user registration button, represents a single touch action by the user from the first time point (T1) to the fourth time point (T4).

[0065] The second threshold time, like the first threshold time, can be set according to the results of learning the user's usage pattern. For example, the second threshold time may be 2 to 3 seconds. The electronic device (101) may display the initial execution screen of the AI ​​assistant as a result of calling the AI ​​assistant. Referring to FIG. 2, the electronic device (101) may output a message saying "Hello, this is the AI ​​assistant" and indicate a waiting state for user query input to the AI ​​assistant. In another embodiment, the electronic device (101) may activate a specific function of the AI ​​assistant as a result of calling the AI ​​assistant. For example, when calling the AI ​​assistant, the electronic device (101) may activate a search function using the AI ​​assistant and output a screen display (e.g., a blur effect) that indicates a waiting state for user input.

[0066] In one embodiment, the electronic device (101) may determine that the touch input is a misrecognition of the user registration button and terminate the visual signal display if the touch input ends before the second threshold time elapses.

[0067] According to one embodiment, the electronic device (101) comprises a display; a sensor; a memory including at least one storage medium including instructions; and at least one processor including a processing circuit; wherein the instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to: detect a first touch input in a first area of ​​the display using the sensor, and in response to identifying that the first touch input persists for a first threshold time and that the shape of the first touch input matches a user registration button, display a visual signal indicating that the user registration button is being recognized in the area where the first touch input is recognized, and in response to the first touch input persisting for a second threshold time or longer, cause the device to execute an AI assistant set on the user registration button.

[0068] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to: identify the first touch input as a touch gesture for the first area in response to the first touch input ending within the first threshold time.

[0069] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to: identify the first touch input as a long-press gesture for the first area in response to identifying that the first touch input lasts for longer than the first threshold time and that the shape of the first touch input does not match the user registration button.

[0070] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to: process the first touch input as a misrecognition in response to the shape of the first touch input matching the user registration button and the first touch input ending within the second threshold time.

[0071] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to determine whether the shape of the first touch input matches the user registration button using a machine learning model that has learned one or more user hand shape data for the user registration button stored in the memory.

[0072] According to one embodiment, the first touch input can be detected by a capacitive touch sensor.

[0073] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to: derive one or more closed curves according to the magnitude of the capacitance from the first touch input, and determine whether the one or more closed curves match the user registration button.

[0074] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the visual display to be gradually emphasized over time in response to the first touch input being continuously detected from the first threshold time to the second threshold time.

[0075] According to one embodiment, when the commands are executed individually or collectively by the at least one processor, the electronic device may be configured such that: the user registration button is configured by a user registration button configuration operation, and the user registration button configuration operation may include the following operations: outputting a user registration button configuration guidance message through the display, receiving a second touch input through the display, setting a certain range of area including the location where the second touch input was received as a first recognition area, displaying the first recognition area on the display, outputting a message requesting additional touch input along the first recognition area, receiving one or more touch inputs on the first recognition area, and storing the second touch input and the one or more touch inputs as one or more user hand shape data for the user registration button.

[0076] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to: acquire a touch video image by the sensor while receiving the second touch input and the one or more touch inputs, extract one or more user hand shape features included in the touch video image, and store them as user hand shape data for each touch input.

[0077] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to: receive one or more user inputs through the display while the first touch input persists after the second threshold time, and in response to the termination of the first touch input, display a screen of the processing result of the AI ​​assistant for the one or more user inputs on at least a part of the display.

[0078] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to: receive a first user input for an execution screen of a first application on the display while the AI ​​assistant is activated, and display a screen of the AI ​​assistant's processing result for the first user input on at least a part of the display.

[0079] FIG. 3 is a flowchart illustrating an AI assistant call operation of an electronic device according to one embodiment of the present disclosure.

[0080] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1) can call an AI assistant by detecting the action of pressing a user registration button that has been previously set and registered by a user. Unlike clicking an icon to call the AI ​​assistant or calling the AI ​​assistant by voice command, the electronic device (101) can immediately call the AI ​​assistant by pressing the user registration button at any location on the screen while using another application, so as to use the AI ​​assistant together. In the following embodiments, each action may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each action may be changed, and at least two actions may be performed in parallel.

[0081] In operation 310, an electronic device (101) according to one embodiment can receive a first touch input on a display.

[0082] In operation 320, the electronic device (101) according to one embodiment can determine whether the first touch input is maintained for a first threshold time or longer. In response to the first touch input ending within the first threshold time, the electronic device (101) can recognize the first touch input as a simple touch gesture (operation 370). If the first touch input is maintained for a first threshold time or longer, the electronic device (101) can determine whether the shape of the first touch input matches a user registration button (operation 330).

[0083] In operation 330, the electronic device (101) according to one embodiment can determine whether the shape of the first touch input matches one or more shapes designated as user registration buttons. If the electronic device (101) determines that the shape of the first touch input does not match the shape of the user registration buttons, it can recognize the first touch input as a simple long press gesture (operation 380).

[0084] In one embodiment, the electronic device (101) can calculate the degree to which a first touch input matches a user registration button using an AI machine learning model, and if the match is higher than a reference accuracy, it can determine that the two shapes match. The AI ​​machine learning model can be trained using touch input data input during the user registration button setting operation process by the user as training data.

[0085] In one embodiment, the first touch input may be detected by a capacitive touch sensor. The electronic device (101) may derive one or more closed curves according to the magnitude of the capacitance from the first touch input and determine whether one or more closed curves correspond to a user registration button. The user registration button may be set to shapes of closed curves having a magnitude greater than or equal to a reference capacitance. The reference capacitance may be determined according to the user's touch pattern. The 330 operation is not necessarily performed after the 320 operation, and the 330 operation may be performed before the 320 operation, and the two operations may be performed simultaneously.

[0086] In operation 340, the electronic device (101) according to one embodiment may display a visual signal indicating that the user registration button is being recognized in an area where the first touch input is recognized. The electronic device (101) may provide feedback to the user that the user registration button is being recognized by continuing to display the visual signal while the first touch input continues.

[0087] In operation 350, the electronic device (101) according to one embodiment can determine whether the first touch input exceeds a second threshold time. The second threshold time may represent the time during which the action of pressing the user registration button is maintained. The display of the electronic device (101) is a touch screen, and touch-based gestures by the user's hand (e.g., tap, double tap, long press) are already mapped as actions to basic interactions. To be clearly distinguished from basic interactions and to reduce the probability of misrecognition by the user's hand, the time during which the first touch input remains in a touch state may be set longer than the time recognized as an existing long press gesture. In response to determining that the first touch input exceeds the second threshold time, the electronic device (101) may execute an AI assistant set on the user registration button. In response to the first touch input ending within the second threshold time, the electronic device (101) may process the first touch input as a misrecognition.

[0088] In 360 operation, an electronic device (101) according to one embodiment may execute an AI assistant when a first touch input is maintained for a second threshold time exceeding the time when the first touch input is first detected. The electronic device (101) may display an initial execution screen of the AI ​​assistant or display an activation state resulting from the execution of the AI ​​assistant on an existing screen.

[0089] FIG. 4 is an example screen of an electronic device setting a user registration button according to one embodiment of the present disclosure.

[0090] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can perform a setting operation for a user registration button.

[0091] In the first screen (410), the electronic device (101) can output a guidance message regarding the user registration button setting. The electronic device (101) can output a message such as, "You can set a user registration button to call the AI ​​assistant. Please input a long touch action with your finger." The electronic device (101) can display a main recognition area (430) for the user registration button. In the first screen (410), the main recognition area (430) is indicated by a dotted line, and a frequently used part may be the main recognition area (430).

[0092] The electronic device (101) can detect a shape (411) resulting from a first touch input on a first screen (410). To recognize the shape of the touch input, the electronic device (101) may include one or more sensors on the display. For example, the display may be a touch screen. The sensors may be, for example, an infrared (IR) sensor, a capacitive touch sensor, a pressure sensor, or an optical sensor. The electronic device (101) may acquire touch video data through the sensors and create an AI machine learning model using the touch video data as training data. When the AI ​​machine learning model receives a touch input, it may determine whether it matches the shape of a user registration button.

[0093] The electronic device (101) can detect the duration of the first touch input. The electronic device (101) can set a touch holding threshold time for the action of pressing the user registration button based on the duration of the first touch input.

[0094] The shape (411) resulting from the first touch input may be defined as one or more contact surface shapes (401) according to the change in the contact surface as the first touch input is pressed. If the touch screen of the electronic device (101) is a capacitive touch sensor, the contact surface shape (401) may represent the capacitance for a single touch. In the contact surface shape (401), closed curves may represent different capacitance values. In the contact surface shape (401), similar to contour lines, the capacitance is higher the closer the closed curve is to the center, and the capacitance of the closed curve is lower the further it is from the center. When the electronic device (101) determines the degree to which the contact surface shape (401) matches the user registration button, it may compare the shape of at least one closed curve among the closed curves of the contact surface shape (401). The electronic device (101) may determine whether it matches the user registration button based on the capacitance value corresponding to at least one closed curve. For example, the electronic device (101) can compare the shape of a user registration button with one or more closed curves that are larger than the first capacitance value corresponding to the first closed curve in the detected contact surface shape (401). The electronic device (101) can compare the closed curves that are larger than the reference capacitance among one or more closed curves constituting the contact surface shape (401) with the user registration button.

[0095] When the electronic device (101) sets the contact surface shape (401) as a user registration button, it can store the shapes of closed curves having a size greater than or equal to the reference capacitance as the user registration button. The contact surface shape (401) for a touch input pressed with a certain force or more can be stored as the user registration button.

[0096] In the second screen (420), the electronic device (101) may output a message requesting additional touch input from the user along the main recognition area (430). The electronic device (101) may output a message such as, "Please input a long touch action with the same finger multiple times along the main recognition area." The electronic device (101) may detect touch inputs (421, 422, 423, 424, 425) that contact the same finger at multiple locations. The touch inputs (421, 422, 423, 424, 425) may have different shapes as the contact surface of the finger varies depending on the touch location. The electronic device (101) may store the shapes of one or more touch inputs for the user registration button. Touch inputs (421, 422, 423, 424, 425) can also have a contact surface that changes depending on the degree of pressing, just like the first touch input (411), and each can be defined as multiple overlapping shapes.

[0097] In one embodiment, the electronic device (101) can set and register a long touch action by multiple fingers as a user registration button. At this time, the user may use the same finger that normally performs a touch gesture on the electronic device (101) (e.g., right index finger or left thumb) as the user registration button to call an AI assistant, or may use a finger that is not normally used (e.g., right ring finger or little finger) as the hand pressing the user registration button. The electronic device (101) can store and manage user registration buttons for various cases by reflecting the user's various usage states.

[0098] FIG. 5 is an example of a user feedback screen for a user registration button of an electronic device according to one embodiment of the present disclosure.

[0099] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can obtain accurate user touch input data through feedback on the setting operation for the user registration button.

[0100] In the first screen (510), the electronic device (101) can output a touch input guidance message for the user registration button. The user can obtain information on how to set up the user registration button from the message "Please register a hand shape in a comfortable position following the area below."

[0101] In the second screen (520), the electronic device (101) may provide a feedback screen to check whether the registered hand shape is properly recognized as the user registration button after setting the user registration button. The electronic device (101) may output a message saying "Please try to recognize the registered finger comfortably" and determine whether the user touch input detected through the screen matches the user registration button.

[0102] In the third screen (530), the electronic device (101) may provide a feedback screen to request additional touch input. The electronic device (101) may request additional touch input for some locations, for example, by displaying a display for some areas of the screen and displaying a message such as "Please register a hand shape again in the area below" to allow the user to conveniently add touch input.

[0103] FIG. 6 is an example of a user registration button by another finger according to an embodiment of the present disclosure.

[0104] According to one embodiment, the user registration button may be defined by a user setting action. The user registration button may be predefined by the user so that it can be used conveniently and quickly to utilize the AI ​​assistant while the user is using the electronic device (101), for example, while using a specific application (e.g., gallery, web browser). The user registration button may be defined in the shape of a part of the hand (e.g., thumb, index finger) that the user intends to use, according to the user's usual usage pattern and preference.

[0105] In the first usage state (610), the electronic device (e.g., the electronic device (101) of FIG. 1) can detect a left-hand grip using one or more sensors (e.g., a capacitive touch sensor, a pressure sensor, a proximity sensor, an inertial sensor (IMU)). For example, the electronic device (101) can determine a left-hand grip if contact is detected at the left edge of the housing. In the left-hand grip state of the electronic device (101), a part of the left hand (e.g., the side of the thumb) (611) can be set and registered as a user registration button.

[0106] In the second usage state (620), the electronic device (101) can detect a right hand grip using one or more sensors and register a part of the right hand (e.g., the side of the thumb) (621) as a user registration button.

[0107] In the third usage state (630), the electronic device (101) can detect a left-hand grip using one or more sensors and register a part of the right hand (e.g., the front of the index finger) (631) as a user registration button.

[0108] The first usage state (610) and the second usage state (620) involve calling the AI ​​assistant using the hand gripping the electronic device (101), and the third usage state (630) involves calling the AI ​​assistant using the hand other than the hand gripping the electronic device (101). Depending on their usage patterns, the user can set an easily accessible hand, finger, and contact surface as a user registration button through a user setting registration process. Alternatively, the user can use a hand, finger, and contact surface that is not frequently used to call the AI ​​assistant by setting a user registration button that is not the hand, finger, and contact surface they commonly use. The user registration button can be personalized to reflect the user's intent.

[0109] FIG. 7 is a flowchart illustrating the search operation of an AI assistant of an electronic device according to one embodiment of the present disclosure.

[0110] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1) can call an AI assistant using a user registration button and use the search function of the AI ​​assistant. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0111] In operation 710, an electronic device (101) according to one embodiment can recognize a user registration button through a display. The electronic device (101) receives a touch input through a display, and if the shape of the touch input matches the user registration button and the touch input is maintained for a certain threshold time or longer, it can identify that the user registration button is recognized.

[0112] In operation 720, the electronic device (101) according to one embodiment can activate the AI ​​assistant set on the user registration button and switch to a user input waiting state.

[0113] In operation 730, the electronic device (101) according to one embodiment can determine whether input to the user registration button is maintained. The electronic device (101) can receive multi-user input if the user registration button is recognized and touch input to the user registration button is maintained even after the AI ​​assistant is activated (operation 742). The electronic device (101) can receive multiple user inputs while the user registration button is pressed and process them as inputs for the AI ​​assistant.

[0114] The electronic device (101) can receive one user input (741) when the user registration button is recognized and the AI ​​assistant is activated, and the touch input to the user registration button is terminated.

[0115] In operation 741, an electronic device (101) according to one embodiment can receive user input as an input value of an AI assistant.

[0116] In operation 742, the electronic device (101) according to one embodiment can receive multi-user input as an input value for the AI ​​assistant while the user registration button is pressed. The electronic device (101) receives multi-user input and, when the touch input for the user registration button ends, can determine whether to select an area to display the execution result of the AI ​​assistant (operation 750).

[0117] In operation 750, the electronic device (101) according to one embodiment may determine whether to select an area to display the execution result of the AI ​​assistant. If the electronic device (101) receives user input drawing a line for a first direction, it may determine that the area of ​​the first direction has been selected as the area to display the execution result of the AI ​​assistant (operation 762). If the electronic device (101) does not receive user input within a set time, it may determine that the area to display the execution result of the AI ​​assistant has not been selected (operation 761).

[0118] In operation 761, the electronic device (101) according to one embodiment may display an AI assistant execution screen in a designated area when no area to display execution results is selected by the user. For example, the electronic device (101) may display an execution screen of an AI assistant that processes user input or multi-user input as a query in the form of a pop-up window overlaid on a part of an existing screen. Alternatively, the electronic device (101) may display an existing screen in a first part of the display and display an execution screen of an AI assistant in a second part separated from the first part.

[0119] 762 In operation, an electronic device (101) according to one embodiment may display an execution screen of an AI assistant that has processed user input or multi-user input as a query in the selected area when the area to display the execution result is selected by the user. The selected area may be an area separated from the existing screen, or an area that overlaps with at least a part of the existing screen.

[0120] FIG. 8 is an example of an AI assistant search function screen using a user registration button of an electronic device according to one embodiment of the present disclosure.

[0121] In one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can use the search function of an AI assistant set on a user registration button.

[0122] In the first screen (810), the electronic device (101) can receive a touch input (801) at any location on the first screen that is output through the display. The electronic device (101) determines that the shape of the touch input (801) matches a user registration button, and if the touch input (801) lasts for a threshold time or longer, it can execute an AI assistant.

[0123] In the second screen (820), the electronic device (101) may display the first screen, which is the existing screen, in a deactivated state and output a display indicating that the AI ​​assistant is running and active. The electronic device (101) may display the first screen dimmed and display the active state of the AI ​​assistant, such as "This is an AI assistant. Please select a search area," superimposed on the first screen.

[0124] In the third screen (830), the electronic device (101) can receive user input selecting a first area (831) of the first screen that is dimmed.

[0125] In the fourth screen (840), when the electronic device (101) receives user input (841) for drawing a line for the first direction, it can identify that a portion of the first direction has been selected.

[0126] In the fifth screen (850), the electronic device (101) can display the search results of the AI ​​assistant in a part of the first direction (851) by overlapping it with a part of the first screen, which is an existing screen, using an object included in the first area as input.

[0127] FIG. 9 is an example of a multi-search function screen of an AI assistant using a user registration button of an electronic device according to one embodiment of the present disclosure.

[0128] In one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can use the multi-search function of the AI ​​assistant by continuously pressing the user registration button.

[0129] In the first screen (910), the electronic device (101) can receive a touch input (901) at any location on the second screen output through the display. The electronic device (101) determines that the shape of the touch input (901) matches a user registration button, and if the touch input (901) lasts for a threshold time or longer, it can execute an AI assistant.

[0130] In the second screen (920), the electronic device (101) may display the second screen, which is the existing screen, in a deactivated state and output an indication that the AI ​​assistant is running and is in an activated state. The electronic device (101) may display the second screen dimmed and display the activation state of the AI ​​assistant, such as "This is the AI ​​assistant. Please select a search area," superimposed on the second screen. The electronic device (101) may detect that touch input to the user registration button continues to be maintained even after the AI ​​assistant is running.

[0131] In the third screen (930), the electronic device (101) can receive user input selecting the first area (931) of the second screen that is dimmed while the user registration button is still pressed. In the fourth screen (940), the electronic device (101) can receive user input selecting the second area (941) that is still dimmed while the user registration button is still pressed. Afterward, the electronic device (101) can detect that the touch input for the user registration button has ended.

[0132] In the fifth screen (950), when the electronic device (101) receives user input (951) for drawing a line for the first direction, it can identify that a portion of the first direction has been selected.

[0133] In the sixth screen (960), the electronic device (101) can display the search results of the AI ​​assistant in a part of the first direction area (961) by overlapping the object included in the first area with the existing screen, the second screen, as input.

[0134] In the case of multi-search in Fig. 9, unlike simple search in Fig. 8, there is a difference in that the user registration button can continuously maintain an active state to receive user input for the AI ​​assistant in addition to the execution of the AI ​​assistant.

[0135] FIG. 10 is an example of a search result screen of an AI assistant in an electronic device according to one embodiment of the present disclosure.

[0136] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can output the search result screen of the AI ​​assistant when the search function of the AI ​​assistant is used during the existing screen display, taking into account the relationship with the existing screen, the horizontal or vertical display state of the electronic device (101), etc.

[0137] The first screen (1010) and the second screen (1020) can display an AI search result screen on an existing screen while the electronic device (101) is displayed vertically. The first screen (1010) displays the AI ​​search result screen superimposed on a part of the existing screen. At this time, the AI ​​search result screen can be displayed with a certain degree of transparency. The electronic device (101) can display the AI ​​search result screen in an area close to the search area on the existing screen. The second screen (1020) can display the AI ​​search result screen in an area separated from the existing screen. The electronic device (101) can reduce the size of the existing screen and output the AI ​​search result screen in a certain area separated from the existing screen.

[0138] The third screen (1030) and the fourth screen (1040) can display an AI search result screen for an existing screen while the electronic device (101) is displayed horizontally. The third screen (1030) is an example in which the AI ​​search result screen is superimposed and displayed on a part of the existing screen, similar to the first screen (1010). The fourth screen (1040) is an example in which the AI ​​search result screen is displayed in an area separated from the existing screen, similar to the second screen (1020).

[0139] According to one embodiment, when an electronic device (101) includes one or more displays (e.g., a flexible main display, a sub-display) by means of a multi-foldable housing, the area where the existing screen and the AI ​​search result screen are displayed can be determined according to the folded state and horizontal / vertical ratio of the multi-foldable.

[0140] FIGS. 11a, 11b, and 11c are examples of use of a user registration button of an electronic device according to one embodiment of the present disclosure.

[0141] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can execute an AI assistant set on a user registration button by pressing the user registration button (touch input) while in use. The electronic device (101) can activate and display a specific function of the AI ​​assistant according to the characteristics of the application being displayed at the point where the user registration button is recognized. For example, referring to FIG. 11a, the electronic device (101) can detect the action of pressing the user registration button through touch input on text while displaying a web page. Referring to FIG. 11b, the electronic device (101) can detect the action of pressing the user registration button through touch input on an object while displaying an image. Referring to FIG. 11c, the electronic device (101) can detect the action of pressing the user registration button through touch input on content while multimedia content is being played.

[0142] In FIG. 11a, the electronic device (101) can display a screen in which different applications are executed in a first area (1110) and a second area (1120), respectively. While the first area (1110) is displaying a web page, the electronic device (101) detects a first touch input (1101) for a first text portion (1111), identifies that the shape of the first touch input (1101) matches a user registration button, and can execute an AI assistant in response to the first touch input (1101) being maintained for a threshold time or longer. The electronic device (101) can select a search function for the first text portion (1111) of the web page among one or more functions of the AI ​​assistant. The electronic device (101) can display the search results for the first text portion (1111) together on the execution screen of the AI ​​assistant. The electronic device (101) can display an AI search result screen (1106) on a part of the display. The electronic device (101) can display a part of the search results on the AI ​​search result screen (1105) and display the text "Search Results" in a bold and underlined font to provide the entire search result. The electronic device (101) can provide the entire search result when the text is clicked.

[0143] The AI ​​assistant may provide at least some of the following functions: text or image search, document analysis, user question and answer, schedule management, translation, document or image creation, provision of various information, or management of other user devices. The electronic device (101) may analyze an object (e.g., text, image, content) at the location where the user registration button is detected and prioritize selecting a function that is highly relevant to that object. Alternatively, the electronic device (101) may select a search function based on the previous user's usage history, such as when the AI ​​assistant is called from a web page and the search function has been frequently used. The electronic device (101) may select the functions of the AI ​​assistant by one or more methods and provide them immediately when called by the user registration button.

[0144] In FIG. 11b, the electronic device (101) may display a screen in which different applications are executed in a first area (1110) and a second area (1120), respectively. While the first area (1110) is displaying a web page containing an image file, the electronic device (101) may detect a second touch input (1102) for a first object (e.g., a table) (1112), identify that the shape of the second touch input (1102) matches a user registration button, and execute an AI assistant in response to the second touch input (1102) being maintained for a threshold time or longer. The electronic device (101) may select a document analysis function with respect to the object of the image among one or more functions of the AI ​​assistant. The electronic device (101) may display on the execution screen (1106) of the AI ​​assistant that it can provide an analysis function for the first object (1112). For example, the electronic device (101) can output a message asking, "Do you need an analysis of the table in the report you are viewing?" and display the keyword "analysis" in bold and underlined font. When the electronic device (101) clicks on the word "analysis," it can use the document analysis function of the AI ​​assistant to analyze the report displayed on the first screen (1110) and provide an analysis result for the first object (1112).

[0145] In FIG. 11c, the electronic device (101) can output a playback screen of multimedia content through a specific application in a third area (1130). The electronic device (101) can detect a third touch input (1103) at any location on the playback screen output in the third area (1130), identify that the shape of the third touch input (1103) matches a user registration button, and, in response to the third touch input (1103) being maintained for a threshold time or longer, execute an AI assistant. The electronic device (101) can select a recommendation function related to multimedia content among one or more functions of the AI ​​assistant. On the execution screen (1107) of the AI ​​assistant, the electronic device (101) can display the word "recommend" in a bold and underlined font along with a message saying, "Hello, this is the AI ​​assistant. How can I help you? Would you like me to recommend a video related to the piano performance you are watching?" When the electronic device (101) clicks the word "recommend," it can use the recommendation function of the AI ​​assistant to provide one or more video lists related to the content being displayed as recommended videos.

[0146] According to one embodiment, the electronic device (101) can expand the user experience by selecting and providing a specific function of an AI assistant to the user based on the location of the screen where the user registration button is detected, object information of the location, and application information. Although not disclosed in the drawings, the electronic device (101) receives a fourth touch input for a function icon displayed on a display screen, and if the fourth touch input corresponds to the action of pressing the user registration button, it can select a function of an AI assistant that is highly related to the function icon and execute the AI ​​assistant.

[0147] FIG. 12 is an example of sharing the execution result of an AI assistant of an electronic device with another electronic device according to one embodiment of the present disclosure.

[0148] According to one embodiment, an electronic device (101) (e.g., the electronic device (101) of FIG. 1) detects a user registration button (1201) and can share the result of running an AI assistant with other electronic devices (e.g., 1210, 1220, 1230, 1240). The other electronic devices may match the user of the electronic device (101). The other electronic devices may be wearable electronic devices that can be worn by the user of the electronic device (101) (e.g., a watch-type wearable device (1210), an earphone-type wearable electronic device (1220)). The other electronic devices may be electronic devices located in close proximity to the electronic device (101) (e.g., a PC, a tablet).

[0149] An electronic device (101) according to one embodiment detects a user registration button (1201), and when a user input (1202) pointing in a specific direction is input, it can share the search results of the AI ​​assistant with another electronic device (e.g., PC (1230)) located in that direction. The electronic device (101) can transmit the search results of the AI ​​assistant to the PC (1230). By sharing the execution results of the AI ​​assistant with other electronic devices, the electronic device (101) can extend the usability of the AI ​​assistant to electronic devices that do not provide the AI ​​assistant.

[0150] The embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" each may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0151] The term “module” as used in the embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0152] One embodiment of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0153] According to one embodiment, the method according to the embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0154] According to one embodiment, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to one embodiment, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to one embodiment, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device, display; Sensor; Memory comprising at least one storage medium for storing instructions; and It includes at least one processor including a processing circuit; When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Using the above sensor, a first touch input is detected in a first area of ​​the display, and In response to identifying that the first touch input persists for a first threshold time or longer and that the shape of the first touch input matches a user registration button, a visual signal indicating that the user registration button is being recognized is displayed in the area where the first touch input is recognized, and An electronic device that causes an AI assistant set on a user registration button to be executed in response to the first touch input being sustained for a second threshold time or longer.

2. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that causes the first touch input to be identified as a touch gesture for the first area in response to the first touch input being terminated within the first threshold time.

3. In Paragraph 1 or 2, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that causes the first touch input to be identified as a long-press gesture for the first area in response to identifying that the first touch input is sustained for longer than the first threshold time and that the shape of the first touch input does not match the user registration button.

4. In any one of paragraphs 1 through 3, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that causes the first touch input to be misrecognized in response to the shape of the first touch input matching the user registration button and the first touch input ending within the second threshold time.

5. In any one of paragraphs 1 through 4, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that causes a machine learning model to learn one or more user hand shape data for the user registration button stored in the memory to determine whether the shape of the first touch input matches the user registration button.

6. In any one of paragraphs 1 through 5, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: The first touch input is detected by a capacitive touch sensor, and One or more closed curves according to the magnitude of the capacitance are derived from the first touch input above, and An electronic device that causes one or more closed curves to match the user registration button.

7. In any one of paragraphs 1 through 6, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that causes the visual display to be gradually emphasized over time in response to the first touch input being continuously detected from the first threshold time to the second threshold time.

8. In any one of paragraphs 1 through 7, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: The above user registration button is set by a user registration button setting operation, and the above user registration button setting operation is, Display a user registration button setting guide message through the above display, and A second touch input is received through the above display, and A certain range of area including the location where the second touch input is received is set as the first recognition area, and The first recognition area is displayed on the display, and Output a message requesting additional touch input along the first recognition area, and Receiving one or more touch inputs on the first recognition area, and An electronic device that causes the second touch input and the one or more touch inputs to be stored as one or more user hand shape data for the user registration button.

9. In any one of paragraphs 1 through 8, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: While receiving the second touch input and the one or more touch inputs, a touch video image is acquired by the sensor, and An electronic device that extracts one or more user hand shape features included in the above touch video image and causes them to be stored as user hand shape data for each touch input.

10. In any one of paragraphs 1 through 9, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: While the first touch input continues after the second threshold time, one or more user inputs are received through the display, and An electronic device that causes a screen of the processing result of the AI ​​assistant for the one or more user inputs to be displayed on at least a part of the display in response to the termination of the first touch input.

11. In any one of paragraphs 1 through 10, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: While the AI ​​assistant is activated, a first user input is received for the execution screen of the first application on the display, and An electronic device that causes the screen of the processing result of the AI ​​assistant for the first user input to be displayed on at least a part of the display.

12. In a method of an electronic device, An operation of detecting a first touch input in a first area of ​​a display using a sensor; An operation of displaying a visual signal indicating that the user button is being recognized in an area where the first touch input is recognized, in response to identifying that the first touch input persists for a first threshold time or longer and that the shape of the first touch input matches a user registration button; and A method comprising an action of executing an AI assistant set on a user registration button in response to the first touch input continuing for a second threshold time or longer.

13. In Paragraph 12, A method comprising an operation of identifying the first touch input as a touch gesture for the first area in response to the first touch input ending within the first threshold time.

14. In Paragraph 12 or 13, A method comprising identifying the first touch input as a long press gesture for the first area in response to identifying that the first touch input persists for longer than the first threshold time and that the shape of the first touch input does not match the user registration button.

15. A non-transitory computer-readable storage medium that records instructions, When the above instructions are executed by one or more processors, the one or more processors: An operation of detecting a first touch input in a first area of ​​a display using a sensor; An operation of displaying a visual signal representing a first application associated with the user registration button in an area where the first touch input is recognized, in response to identifying that the first touch input persists for a first threshold time or longer and that the shape of the first touch input matches the user registration button; and A storage medium that performs an operation of displaying an execution screen of the first application on at least a part of the display in response to the first touch input continuing for a second threshold time or longer.

Images