Button module capable of receiving input of user gesture, and electronic device comprising same

Abstract

This electronic device comprises: a housing; a button provided on a side surface of the housing; a memory, which stores instructions and includes one or more pieces of storage media; a printed circuit board including a first pressure sensor, which senses pressure applied by a first pressing part of the button provided on the side surface of the housing of the electronic device, and a second pressure sensor, which senses pressure applied by a second pressing part of the button; and at least one processor including processing circuitry, wherein, when executed by the at least one processor, the instructions individually or collectively instruct the electronic device to: perform an operation corresponding to the first pressing part when pressure greater than or equal to a first threshold value is sensed by the first pressure sensor; perform an operation corresponding to the second pressing part when pressure greater than or equal to the first threshold value is sensed by the second pressure sensor; receive sensing data from the first pressure sensor and the second pressure sensor at preset time intervals when pressure between the first threshold value and the second threshold value is sensed by the first pressure sensor or the second pressure sensor; input the sensing data into an artificial intelligence model so as to identify a user gesture on the basis of movement of a position at which the pressure is sensed on the button; and perform an operation corresponding to the user gesture.

Description

A button module capable of receiving user gesture input and an electronic device including the same

[0001] The present disclosure relates to a button module capable of receiving user gestures and an electronic device including the same.

[0002] Electronic devices that input user commands via buttons are being developed. For example, smartphones are a prime example. These devices can perform specific functions when a button is pressed.

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0004] According to one embodiment of the present disclosure, an electronic device comprises a housing, a button provided on a side of the housing, a memory including one or more storage media for storing instructions, a printed circuit board including a first pressure sensor for detecting pressure applied through a first pressing portion of a button provided on a side of the housing of the electronic device and a second pressure sensor for detecting pressure applied through a second pressing portion of the button, and at least one processor including a processing circuit, wherein when the instructions are executed individually or collectively by the at least one processor, the electronic device performs an operation corresponding to the first pressing portion when a pressure greater than or equal to a first threshold value is detected at the first pressure sensor, performs an operation corresponding to the second pressing portion when a pressure greater than or equal to the first threshold value is detected at the second pressure sensor, and when a pressure between the first threshold value and the second threshold value is detected at the first pressure sensor or the second pressure sensor, receives detection data from the first pressure sensor and the second pressure sensor at a preset time interval, and the detection data is artificial intelligence It can be configured to input into the model to identify a user gesture based on the movement of the position where the pressure is detected on the button, and to perform an action corresponding to the user gesture.

[0005] A method of operation of an electronic device including a first pressing part and a second pressing part according to an embodiment of the present disclosure, and a button provided on a side, may include: an operation of performing an operation corresponding to the first pressing part when a pressure greater than or equal to a first threshold value is detected by a first pressure sensor that detects pressure applied through the first pressing part; an operation of performing an operation corresponding to the second pressing part when a pressure greater than or equal to the first threshold value is detected by a second pressure sensor that detects pressure applied through the second pressing part; an operation of receiving detection data from the first pressure sensor and the second pressure sensor at a preset time interval when a pressure between the first threshold value and the second threshold value is detected by the first pressure sensor or the second pressure sensor; an operation of inputting the detection data into an artificial intelligence model to identify a user gesture based on the movement of the position where the pressure is detected on the button; and an operation of performing an operation corresponding to the user gesture.

[0006] A computer-readable recording medium comprising a program for executing a method of operation of an electronic device including a first pressing part and a second pressing part according to an embodiment of the present disclosure and a button provided on a side, wherein the method of operation of the electronic device may include: an operation of performing an operation corresponding to the first pressing part when a pressure greater than or equal to a first threshold value is detected by a first pressure sensor detecting pressure applied through the first pressing part; an operation of performing an operation corresponding to the second pressing part when a pressure greater than or equal to the first threshold value is detected by a second pressure sensor detecting pressure applied through the second pressing part; an operation of receiving detection data from the first pressure sensor and the second pressure sensor at a preset time interval when a pressure between the first threshold value and the second threshold value is detected by the first pressure sensor or the second pressure sensor; an operation of inputting the detection data into an artificial intelligence model to identify a user gesture based on the movement of the position where the pressure is detected on the button; and an operation of performing an operation corresponding to the user gesture.

[0007] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0008] FIG. 1 is a block diagram of an exemplary electronic device capable of performing the operations described in the present disclosure.

[0009] FIG. 2 is a perspective view showing a bar-type smartphone as an electronic device according to an embodiment of the present disclosure.

[0010] FIG. 3 is a drawing showing a button of a button module exposed to the outside of a housing of an electronic device according to an embodiment of the present disclosure.

[0011] FIG. 4 is a drawing showing a button module disposed on the side of a housing of an electronic device according to an embodiment of the present disclosure.

[0012] FIG. 5 is a drawing showing a button of an electronic device divided into a plurality of regions according to an embodiment of the present disclosure.

[0013] FIG. 6 is a drawing showing an electronic device that identifies a user gesture based on the movement of a pressure-sensing position according to an embodiment of the present disclosure.

[0014] FIG. 7 is a drawing for illustrating the pressure between a first threshold value and a second threshold value according to an embodiment of the present disclosure.

[0015] FIG. 8 is a diagram illustrating an artificial intelligence model for identifying user gestures according to an embodiment of the present disclosure.

[0016] FIG. 9 is a diagram illustrating training data of an artificial intelligence model according to an embodiment of the present disclosure.

[0017] FIG. 10 is a drawing for explaining UI and haptic feedback according to user gestures according to an embodiment of the present disclosure.

[0018] FIG. 11 is a drawing for explaining a UI according to a user gesture according to an embodiment of the present disclosure.

[0019] The present disclosure will be described in detail below with reference to the attached drawings.

[0020] The terms used in the embodiments of this disclosure have been selected to be as widely used as possible, taking into account their functions within this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant explanatory section of this disclosure. Therefore, terms used in this disclosure should be defined not merely by their names, but based on their meanings and the overall content of this disclosure.

[0021] In this specification, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of such features (e.g., numerical values, functions, operations, or components such as parts) and do not exclude the presence of additional features.

[0022] The expression "at least one of A or / and B" should be understood as representing either "A" or "B" or "A and B".

[0023] Expressions such as "first," "second," "first," or "second" used in this specification may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit said components.

[0024] Where it is stated that a component (e.g., Component 1) is "(operatively or communicatively) coupled with / to" or "connected to" another component (e.g., Component 2), it should be understood that the component may be directly connected to the other component or connected through the other component (e.g., Component 3).

[0025] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "consisting of" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0026] In this specification, the term "user" may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0027] Embodiments of the present disclosure will be described in more detail below with reference to the attached drawings.

[0028]

[0029] FIG. 1 is a block diagram of an exemplary electronic device capable of performing the operations described in the present disclosure.

[0030] Referring to FIG. 1, the electronic device (100) may be one of various forms of electronic devices, such as a notebook (190), smartphones (191) having various form factors (e.g., a bar-type smartphone (191-1), a foldable-type smartphone (191-2), or a sliderable (or rollable)-type smartphone (191-3)), a tablet (192), a wearable device (e.g., a smart watch (193), augmented reality glasses (194)), a cellular phone (not shown), and other similar computing devices (not shown). The components, their relationships, and their functions illustrated in FIG. 1 are illustrative only and are not intended to limit the implementations described or claimed herein. The electronic device (100) may be referred to as a mobile device, a user device, a multifunction device, a portable device, or a server.

[0031] The electronic device (100) may include components including at least one processor (110), at least one memory (120) (hereinafter referred to as memory (120)), at least one display (140) (hereinafter referred to as display (140)), at least one image sensor (150) (hereinafter referred to as image sensor (150)), at least one communication circuit (160) (hereinafter referred to as communication circuit (160)), and / or at least one sensor (170) (hereinafter referred to as sensor (170)), a microcontroller unit (MCU) (131), a haptic driver integrated circuitry (133), and a power management integrated circuitry (PMIC) (180). The components are merely exemplary. For example, the electronic device (100) may include other components (e.g., audio processing circuitry, audio output module, antenna, rechargeable battery, or input / output interface). For example, some components may be omitted from the electronic device (100). For example, some components may be integrated into a single component.

[0032] At least one processor (110) may be implemented as one or more IC (integrated circuit (or circuitry)) chips and may perform various data processing operations. At least one processor (110) may include at least one electrical circuit and may process instructions (or programs, data, etc.) stored in memory (120) individually or collectively in a distributed manner. At least one processor (110) may include a processor assembly including one or more processing circuits. At least one processor (110) may include any processing circuit that is operative to control the performance and operation of one or more components of an electronic device (100) (e.g., memory (120), MCU (131), current driver IC (133), vibration actuator (135), display (140), image sensor (150), communication circuit (160), sensor (170) and / or PMIC (180)). For example, at least one processor (110) (e.g., an application processor (AP)) may be implemented as a system on chip (SoC) (e.g., a single chip or a chipset). For example, at least one processor (110) may be implemented as a plurality of cores (or at least one core circuit), a plurality of chips, or a plurality of chipsets. For example, at least one processor (110) may include one or more processing circuits. For example, at least one processor (110) may include one or more processing circuits configured to perform the various functions of the present disclosure individually and / or collectively.As a non-limiting example, at least a portion of at least one processor (110) may be included in a first chip of an electronic device (100), and at least another portion of at least one processor (110) may be included in a second chip of an electronic device (100) different from the first chip of the electronic device (100).

[0033] For example, at least one processor (110) may include a central processing unit (111), a graphics processing unit (112), a neural processing unit (113), an image signal processor (114), a display controller (115), a memory controller (116), a storage controller (117), a communication processor (118), and / or a sensor interface (119). These components of at least one processor (110) are merely exemplary. For example, at least one processor (110) may include other components. For example, some components of at least one processor (110) may be omitted from at least one processor (110). For example, some components of at least one processor (110) may be included as separate components of the electronic device (100) outside of at least one processor (110). For example, some components of at least one processor (110) (e.g., memory controller (116)) may be included in other components (e.g., at least part of memory (120), an interface (e.g. available for connection to at least one component of the electronic device (100), a display (140) and / or an image sensor (150)).

[0034] At least one processor (110) may cause other components of the electronic device (100) to perform various operations by executing instructions stored in memory (120). A CPU (111) (or central processing circuit) may be configured to control components of at least one processor (110) based on the execution of instructions stored in memory (120) (e.g., volatile memory (121) and / or non-volatile memory (122)). A GPU (112) (or graphics processing circuit) may be configured to execute parallel operations (e.g., rendering). An NPU (113) (or neural processing circuit, or AI (artificial intelligence) chip) may be configured to execute operations for an artificial intelligence model (e.g., convolution computation). An ISP (114) (or image signal processing circuit) may be configured to process a raw image acquired through an image sensor (150) into a format suitable for a component within the electronic device (100) or for a component of at least one processor (110). A display controller (115) (or display control circuit, or DPU (display processing unit)) may be configured to process an image acquired from a CPU (111), GPU (112), ISP (114), or memory (120) (e.g., volatile memory (121)) into a format suitable for a display (140). A memory controller (116) (or memory control circuit) may be configured to control reading data from the volatile memory (121) and writing data to the volatile memory (121). A storage controller (117) (or storage control circuit) may be configured to control reading data from the non-volatile memory (122) and writing data to the non-volatile memory (122).The CP (118) (communication processing circuit) may be configured to process data obtained from a component of at least one processor (110) into a format suitable for transmitting to another electronic device via the communication circuit (160), or to process data obtained from another electronic device via the communication circuit (160) into a format suitable for processing by a component of at least one processor (110). For example, the communication circuit (160) may include one or more communication circuits. The sensor interface (119) (or sensing data processing circuit, sensor hub) may be configured to process data regarding the state of the electronic device (100) and / or the state around the electronic device (100), obtained through the sensor (170), into a format suitable for a component of at least one processor (110).

[0035] Memory (120) may include one or more storage media (or one or more storage devices). For example, memory (120) may include a memory assembly comprising one or more storage media. For example, the one or more storage media may include a hard drive, a permanent memory such as flash memory, read-only memory (ROM) (e.g., non-volatile memory (122)), a semi-permanent memory such as random access memory (RAM) (e.g., volatile memory (121)), any other suitable type of storage (or storage assembly), or any combination thereof. Memory (120) may include a cache memory, which is one or more different types of memory used to temporarily store data for a function or feature of the electronic device (100). As an example not limited to, the cache memory may be included within at least one processor (110). The memory (120) may be fixedly embedded within the electronic device (100) or incorporated into one or more suitable types of components (e.g., a SIM (subscriber identity module) card and / or an SD (secure digital) card) that can be repeatedly inserted into and removed from the electronic device (100).

[0036] For example, memory (120) may store one or more software applications, such as operating system (or system) software applications, firmware software applications, driver software applications, plugin (e.g., add-in, add-on, and / or applet) software applications, and / or any other suitable software applications. For example, the one or more software applications may include instructions executable by at least one processor (110). For example, memory (120) may store instructions that can be called by an application programming interface (API). For example, memory (120) may store instructions within a library.

[0037] The microcontroller unit (131) can control the sensor (170) and input / output device and perform system management tasks in a low-power state. For example, the MCU (130) can process and control sensor data obtained through an accelerometer, gyroscope, and temperature sensor. The MCU (130) can control buttons (210 in FIG. 2), a display (140) (e.g., a touchscreen), and a camera module provided in the electronic device (100). The MCU (130) can perform battery management, power control, and system initialization tasks. The current driver IC (133) can be controlled by the processor (110) or the MCU (131) to drive the vibration actuator (135). The current driver IC (133) may be referred to as the 'vibration driver IC'. The vibration actuator (135) is controlled by a current driver IC (133) and can provide physical feedback by generating vibrations according to one or more vibration patterns. In this disclosure, the current driver IC (133) may be referred to as a vibration driver IC (vibration driver integrated circuitry).

[0038] The PMIC (180) can perform power management of the electronic device (100). For example, the PMIC (180) can perform power distribution, power conversion, power consumption optimization, battery management, power sequencing, and / or protection functions. According to one embodiment, the PMIC (180) can convert a sensing value (analog data) obtained through the sensor (170) in a low-current standby state into a vibration pattern (digital data) on behalf of the processor (110) or MCU (131).

[0039]

[0040] FIG. 2 is a perspective view showing a bar-type smartphone as an electronic device according to an embodiment of the present disclosure. FIG. 3 is a drawing showing a button of a button module exposed to the outside of the housing of an electronic device according to an embodiment of the present disclosure.

[0041] Referring to FIGS. 2 and 3, the electronic device (100) can be implemented as a bar-type smartphone (191-1, see FIG. 1). For example, the button module (200) of the electronic device (100) can be applied to a bar-type smartphone (191-1, see FIG. 1), but is not limited thereto. For example, the button module (200) can be applied to a laptop (190) shown in FIG. 1, a foldable-type smartphone (191-2), a sliderable (or rollable)-type smartphone (191-3), a tablet (192), a smart watch (193), and augmented reality glasses (194).

[0042] Referring to FIGS. 2 and 3, the electronic device (100) may include a housing (195), a display (140) that may be placed on the front of the housing (195), and a button module (200) provided on one side of the housing (195).

[0043] The button module (200) may include at least one button (210) (hereinafter referred to as button (210)) that can be exposed to the outside of the housing (195) so as to be pressed by a user's finger. The button (210) may be positioned along the longitudinal direction (e.g., z-axis direction) of the housing (195). For example, the vertical length (e.g., length in the z-axis direction) of the button (210) may be greater than the horizontal length (e.g., length in the y-axis direction).

[0044] The button (210) may be configured to be visible as a structure segmented into at least two parts. For example, the button (210) may be visible as a structure segmented into a first pressing part (213a) and a second pressing part (213b).

[0045] For example, the first pressing part (213a) may be composed of a first metal member, and the second pressing part (213b) may be composed of a second metal member. However, this is not limited thereto, and the button (210) may include a structure formed integrally (e.g., a single metal member) in which the first pressing part (213a) and the second pressing part (213b) are not physically cut and separated into separate metal members (e.g., a first metal member and a second metal member).

[0046] The first pressing part (213a) and the second pressing part (213b) of the button (210) can be provided along the z-axis direction.

[0047] According to the embodiment, the first metal member and the second metal member constituting the first pressing part (213a) and the second pressing part (213b), respectively, may be made of the same material as the housing (195).

[0048] According to an embodiment, the electronic device (100) can perform a corresponding operation when the first pressing part (213a) or the second pressing part (213b) is pressed by a user.

[0049] For example, the button (210) can be used as a volume control button having the function of adjusting the volume, and the electronic device (100) can increase the volume when the first pressing part (213a) constituting the button (210) is pressed, and can decrease the volume when the second pressing part (213b) is pressed.

[0050] For example, the electronic device (100) can perform an operation corresponding to the first pressing part (213a) (e.g., an operation to increase the volume) if the pressure applied by the user through the first pressing part (213a) is greater than or equal to the first threshold value, and can perform an operation corresponding to the second pressing part (213b) (e.g., an operation to decrease the volume) if the pressure applied by the user through the second pressing part (213b) is greater than or equal to the first threshold value.

[0051] According to an embodiment, the electronic device (100) can detect a swipe (or scroll) motion of a user while a part of the body (e.g., finger (1)) is in contact with the button (210), and can perform a corresponding motion. For example, if the electronic device (100) identifies a user gesture of swiping the button (210) rather than a user input of pressing the first pressing part (213a) or the second pressing part (213b) constituting the button (210), it can perform a motion corresponding to the user gesture.

[0052] For example, the electronic device (100) can identify a user gesture of swiping the button (210) if the pressure applied by the user through the first pressing part (213a) is less than a first threshold value and greater than or equal to a second threshold value (i.e., between the first threshold value and the second threshold value). For example, the electronic device (100) can identify a user swipe gesture in which one area of ​​the first pressing part (213a) is the starting point and one area of ​​the second pressing part (213b) is the ending point.

[0053] For example, the electronic device (100) detects pressure between a first threshold value and a second threshold value in one area of ​​the first pressing part (213a), and when the area where pressure is detected moves from one area of ​​the first pressing part (213a) to one area of ​​the second pressing part (213b), the electronic device (100) can identify a user's swipe gesture that starts at the first pressing part (213a) and stops at the second pressing part (213b).

[0054] Additionally, the electronic device (100) can identify a user gesture of swiping the button (210) if the pressure applied by the user through the second pressing part (213b) is between a first threshold value and a second threshold value. For example, the electronic device (100) can identify a user swipe gesture in which one area of ​​the second pressing part (213b) is the starting point and one area of ​​the first pressing part (213a) is the ending point. For example, the electronic device (100) can identify a user swipe gesture that starts at the second pressing part (213b) and stops at the first pressing part (213a). According to an embodiment, a method for the electronic device (100) to detect the intensity of the pressure and a method for detecting the direction in which the pressure-sensing area moves will be described later.

[0055]

[0056] FIG. 4 is a drawing showing a button module disposed on the side of a housing of an electronic device according to an embodiment of the present disclosure.

[0057] Referring to FIG. 4, the button module (200) may be supported by a structure (196) provided inside the housing (195). Most of the components included in the button module (200) may be located inside the housing (195). The button (210) included in the button module (200) may protrude a predetermined distance from the outer surface of the housing (195). The user may determine the location of the button (210) through sight and / or touch.

[0058] According to an embodiment, the button module (200) may include a button (210), a first pressure sensor (170a) that generates a first pressure signal when pressed by the button (210), a second pressure sensor (170b) that generates a second pressure signal, and a printed circuit board (e.g., a flexible printed circuit board) on which the first pressure sensor (170a) and the second pressure sensor (170b) are placed.

[0059] According to an embodiment, the button (210) may include a first pressing part (213a) corresponding to a first pressure sensor (170a) and a second pressing part (213b) corresponding to a second pressure sensor (170b).

[0060] According to an embodiment, the first pressure sensor (170a) can detect pressure on the first pressing part (213a) and generate a first pressure signal and transmit it to at least one processor (110), and the second pressure sensor (170b) can detect pressure on the second pressing part (213b) and generate a second pressure signal and transmit it to at least one processor (110).

[0061] However, it is not limited thereto, and according to the embodiment, the button (210) may include a single pressing part, and the first pressure sensor (170a) may detect pressure on the upper part of the single pressing part (e.g., the single pressing part is divided into two areas based on the center, and the upper part of the two areas), and the second pressure sensor (170b) may detect pressure on the lower part of the single pressing part.

[0062] According to an embodiment, when the first pressing part (213a) is pressed and the first pressure sensor (170a) detects a pressure greater than or equal to the first threshold value, the first pressure sensor (170a) can generate a first pressure signal corresponding to the pressure greater than or equal to the first threshold value.

[0063] According to an embodiment, the electronic device (100) identifies that the first pressing part (213a) is pressed based on a first pressure signal corresponding to a pressure greater than or equal to a first threshold value, and can perform an operation corresponding to the first pressing part (213a) (e.g., an operation to increase volume).

[0064] According to an embodiment, when the second pressing part (213b) is pressed and the second pressure sensor (170b) detects a pressure greater than or equal to the first threshold value, the second pressure sensor (170b) can generate a second pressure signal corresponding to the pressure greater than or equal to the first threshold value.

[0065] According to an embodiment, the electronic device (100) identifies that the second pressing part (213b) is pressed based on a second pressure signal corresponding to a pressure greater than or equal to a first threshold value, and can perform an operation corresponding to the second pressing part (213b) (e.g., an operation to reduce volume).

[0066] According to an embodiment, the electronic device (100) can identify a user gesture of swiping the button (210) in addition to user input of pressing the first pressing part (213a) or the second pressing part (213b). For example, if the electronic device (100) detects pressure below a first threshold value in either the first pressing part (213a) or the second pressing part (213b), it can identify a user gesture of swiping the button (210) and perform an action corresponding to the user gesture.

[0067]

[0068] FIG. 5 is a drawing showing a button of an electronic device divided into a plurality of regions according to an embodiment of the present disclosure.

[0069] According to an embodiment, the button (210) may be divided into a plurality of regions. For example, the button (210) may include regions corresponding to the first pressing portion (213a) and regions corresponding to the second pressing portion (213b). For example, the button (210) may be divided into the first region (10-1) to the eighth region (10-8).

[0070] Referring to FIG. 5, the first pressing portion (213a) may be divided into a first region (10-1) to a fourth region (10-4). However, this is an example for convenience of explanation and is not limited thereto. For example, the first pressing portion (213a) may be divided into three regions.

[0071] According to an embodiment, when pressure is applied to any one of the regions corresponding to the first pressing part (213a) (e.g., first region (10-1) to fourth region (10-4)), the first pressure sensor (170a) can detect the pressure applied to the first pressing part (213a).

[0072] For example, if a pressure greater than a first threshold value is detected in any one of the areas corresponding to the first pressing part (213a) through the first pressure sensor (170a), the electronic device (100) can identify that the first pressing part (213a) is pressed.

[0073] Referring to FIG. 5, the second pressing portion (213b) may be divided into a fifth region (10-5) to an eighth region (10-8). However, this is an example for convenience of explanation and is not limited thereto. For example, the second pressing portion (213b) may be divided into three regions.

[0074] According to an embodiment, when pressure is applied to any one of the regions corresponding to the second pressing part (213b) (e.g., the fifth region (10-5) to the eighth region (10-8)), the second pressure sensor (170b) can detect the pressure applied to the second pressing part (213b).

[0075] For example, if a pressure greater than the first threshold value is detected in any one of the areas corresponding to the second pressing part (213b) through the second pressure sensor (170b), the electronic device (100) can identify that the second pressing part (213b) is pressed.

[0076] For example, when any one of the first area (10-1) to the fourth area (10-4) included in the first pressing portion (213a) of the button (210) is pressed by a part of the user's body (e.g., the user's finger (1)), if pressure exceeding a predetermined pressure threshold (e.g., a first threshold value) is applied to the first pressure sensor (170a), a first pressure signal may be generated by the first pressure sensor (170a). The electronic device (100) may identify the user input as a first pressing gesture (or a first press gesture) based on the first pressure signal. The electronic device (100) may perform an action corresponding to the first pressing gesture.

[0077] For example, the electronic device (100) can perform an operation to increase the system sound volume of the electronic device (100) or an operation to increase the brightness of the screen of the display (140) of the electronic device (100) based on the first pressure signal.

[0078] For example, when any one of the fifth region (10-5) to the eighth region (10-8) included in the second pressing portion (213b) of the button (210) is pressed by the user's finger (1) of the electronic device (100), if pressure exceeding a predetermined pressure threshold is applied to the second pressure sensor (170b), a second pressure signal may be generated by the second pressure sensor (170b). The electronic device (100) may identify the user input as a second pressing gesture (or a second press gesture) based on the second pressure signal. The electronic device (100) may perform an action corresponding to the second pressing gesture.

[0079] For example, the electronic device (100) may perform an operation to reduce the system sound volume of the electronic device (100) or to reduce the brightness of the screen of the display (140) of the electronic device (100) based on a second pressure signal.

[0080] The subject pressing the first area (10-1) to the eighth area (10-8) of the button (210) is not limited to a part of the user's body. For example, when pressure is detected as the user presses any one of the first area (10-1) to the eighth area (10-8) of the button (210) using a separate tool, the first pressure sensor (170a) may generate a first pressure signal or the second pressure sensor (170b) may generate a second pressure signal.

[0081] According to an embodiment, when the electronic device (100) detects a pressure between a first threshold value and a second threshold value at the first pressure sensor (170a) or the second pressure sensor (170b), it may be identified that the first pressing part (213a) or the second pressing part (213b) is not pressed (e.g., the first pressing gesture or the second pressing gesture is not identified), but that a part of the user's body (e.g., finger (1)) is in contact with the button (210).

[0082] According to an embodiment, when a pressure between a first threshold value and a second threshold value is detected at a first pressure sensor (170a) or when a pressure between a first threshold value and a second threshold value is detected at a second pressure sensor (170b), the electronic device (100) can receive detection data from the first pressure sensor (170a) and the second pressure sensor (170b) at preset time intervals.

[0083] According to an embodiment, the electronic device (100) can identify movement of a part of a user's body (e.g., finger (1)) on the button (210) that has come into contact with the button (210) based on detection data received at preset time intervals. A detailed explanation thereof will be provided with reference to FIGS. 6 to 8.

[0084]

[0085] FIG. 6 is a drawing showing an electronic device that identifies a user gesture based on the movement of a pressure-sensing position according to an embodiment of the present disclosure.

[0086] Referring to FIG. 6, the electronic device (100) can identify that a part of the user's body has come into contact with any one of the first area (10-1) to the fourth area (10-4) included in the first pressing part (213a) when pressure between the first threshold value and the second threshold value is detected through the first pressure sensor (170a).

[0087] According to an embodiment, when the electronic device (100) detects a pressure between a first threshold value and a second threshold value through a first pressure sensor (170a), it can receive detection data from the first pressure sensor (170a) and the second pressure sensor (170b) at preset time intervals.

[0088] According to an embodiment, the electronic device (100) can input detection data received at preset time intervals into an artificial intelligence model to identify the direction of movement of the location where pressure between a first threshold value and a second threshold value is detected on the button (210). For example, the electronic device (100) can input detection data into an artificial intelligence model to identify that the movement of the location where pressure is detected on the button (210) corresponds to a first direction. For example, the first direction may correspond to the direction of movement from the first pressing part (213a) to the second pressing part (213b).

[0089] For example, as illustrated in FIG. 6, when a part of a user's body (e.g., finger (1)) contacts a second area (10-2) among a plurality of areas on a button (210) and a pressure below a first threshold value and above a second threshold value is detected by a first pressure sensor (170a), the electronic device (100) can receive detection data from the first pressure sensor (170a) and the second pressure sensor (170b) at preset time intervals.

[0090] According to an embodiment, the electronic device (100) can identify the movement of the area where pressure is detected between a first threshold value and a second threshold value based on detection data of a time series received at preset time intervals.

[0091] According to an embodiment, the electronic device (100) can identify a swipe gesture in which a part of the user’s body in contact with the button (210) moves in a first direction after the first pressure sensor (170a) detects a pressure between a first threshold value and a second threshold value, and the second pressure sensor (170b) detects a pressure between a first threshold value and a second threshold value. For example, the electronic device (100) can identify a swipe gesture in which the second area (10-2) of the first pressing part (213a) is the starting point and the sixth area (10-6) of the second pressing part (213b) is the ending point.

[0092] However, this is an example for convenience of explanation and is not limited thereto. For example, when the electronic device (100) detects a pressure between a first threshold value and a second threshold value through the second pressure sensor (170b), it may identify that a part of the user's body has come into contact with any one of the fifth area (10-5) to the eighth area (10-8) included in the second pressing part (213b). According to an embodiment, the electronic device (100) may receive detection data from the first pressure sensor (170a) and the second pressure sensor (170b) at preset time intervals.

[0093] According to an embodiment, the electronic device (100) can identify a user's swipe gesture in which a part of the user's body in contact with the button (210) moves in a second direction based on detection data of a time series received at preset time intervals, after the pressure between the first threshold value and the second threshold value is detected at the second pressure sensor (170b) and the pressure between the first threshold value and the second threshold value is detected at the first pressure sensor (170a). For example, the second direction may correspond to the direction of movement from the second pressing part (213b) to the first pressing part (213a).

[0094] For example, the electronic device (100) can identify a swipe gesture in which the sixth area (10-6) of the second pressing part (213b) is the starting point and the second area (10-2) of the first pressing part (213a) is the ending point.

[0095]

[0096] FIG. 7 is a drawing for illustrating the pressure between a first threshold value and a second threshold value according to an embodiment of the present disclosure.

[0097] FIG. 7 illustrates an example of time series detection data received by an electronic device (100) from a first pressure sensor (170a) and a second pressure sensor (170b) at preset time intervals.

[0098] Referring to FIG. 7, when a pressure below a second threshold value is detected by the first pressure sensor (170a), the electronic device (100) can determine that the pressure below the second threshold value is noise.

[0099] According to an embodiment, when the electronic device (100) detects a pressure below a first threshold value and above a second threshold value at the first pressure sensor (170a), it can identify that a finger (1) has come into contact with any one of the areas (e.g., first area (10-1) to fourth area (10-4)) included in the first pressing part (213a) corresponding to the first pressure sensor (170a).

[0100] For example, when a finger (1) contacts a first area (10-1) or a second area (10-2) adjacent to the first pressure sensor (170a) among the areas included in the first pressing part (213a), the pressure detected by the first pressure sensor (170a) may be relatively close to a first threshold value, and when a finger (1) contacts a third area (10-3) or a fourth area (10-4), the pressure detected by the first pressure sensor (170a) may be relatively close to a second threshold value.

[0101] According to an embodiment, the electronic device (100) can identify any one of the first areas (10-1) to fourth areas (10-4) included in the first pressing part (213a) that the finger (1) has contacted, based on the intensity of the pressure detected through the first pressure sensor (170a).

[0102] Referring to the left graph of FIG. 7, the pressure detected by the first pressure sensor (170a) according to the time series detection data received from the first pressure sensor (170a) gradually increases to have a strength between the first threshold value and the second threshold value, and gradually decreases to have a strength less than the second threshold value.

[0103] For example, as the finger (1) in contact with the first pressing part (213a) moves away from the first pressure sensor (170a), the intensity of the pressure detected by the first pressure sensor (170a) gradually decreases, and the electronic device (100) can detect the direction of movement of the finger (1) on the first pressing part (213a) of the button (210) based on the change in the intensity of the pressure detected through the first pressure sensor (170a).

[0104] Referring to the right graph of FIG. 7, the pressure detected by the second pressure sensor (170b) according to the time series detection data received from the second pressure sensor (170b) gradually increases to have a strength between the first threshold value and the second threshold value, and gradually decreases to have a strength less than the second threshold value.

[0105] For example, since the electronic device (100) detects pressure between a first threshold and a second threshold through the first pressure sensor (170a) at time T1 and detects pressure between a first threshold and a second threshold through the second pressure sensor (170b) at time Tn, the electronic device (100) identifies that the user's finger (1) in contact with the button (210) has moved from the first pressing part (213a) to the second pressing part (213b), and can identify a swipe gesture in the first direction based on the movement of the pressure-sensing position.

[0106] However, it is not limited to this, and the electronic device (100) may also identify a swipe gesture in the reverse direction of the first direction (hereinafter, the second direction) based on the detection data of the time series.

[0107] For example, the electronic device (100) can identify a swipe gesture in a second direction, in which one of the multiple areas (e.g., the 6th area (10-6)) is the starting point and the other area (e.g., the 2nd area (10-2)) is the ending point, based on time series detection data, when pressure having a strength between a first threshold value and a second threshold value is first detected in one of the multiple areas (e.g., the 6th area (10-6)), and after the position where the pressure is detected moves in the second direction, pressure is last detected in another of the multiple areas (e.g., the 2nd area (10-2)).

[0108] In the above-described embodiment, the electronic device (100) is described as identifying rule-based user gestures, but is not limited thereto, and the electronic device (100) can identify user gestures by inputting time-series detection data into an artificial intelligence model.

[0109]

[0110] FIG. 8 is a diagram illustrating an artificial intelligence model for identifying user gestures according to an embodiment of the present disclosure.

[0111] Referring to FIG. 8, when a pressure between a first threshold value and a second threshold value is detected at either the first pressure sensor (170a) or the second pressure sensor (170b), the electronic device (100) can receive detection data from the first pressure sensor (170a) and the second pressure sensor (170b) at preset time intervals.

[0112] According to an embodiment, the electronic device (100) can input detection data of a time series received at preset time intervals (e.g., 10 msec intervals) into an artificial intelligence model (1000).

[0113] In the example described above, the first pressure sensor (170a) and the second pressure sensor (170b) were assumed and explained, but it is understood that this is not limited thereto. For example, the electronic device (100) may include the first pressure sensor (170a) to the nth pressure sensor, receive detection data from each of the first pressure sensor (170a) to the nth pressure sensor at preset intervals, and input the time-series detection data into the artificial intelligence model (1000).

[0114] According to an embodiment, the electronic device (100) inputs the sensing data into an artificial intelligence model (1000) to identify the movement of the location where pressure between a first threshold value and a second threshold value is detected.

[0115] According to an embodiment, the artificial intelligence model (1000) may be a model trained to identify the point where contact of the finger (1) began and ended based on time series detection data, and to identify the direction in which the finger (1) moved according to the beginning point and the end point.

[0116] For example, the artificial intelligence model (1000) may be a model trained to identify the initial contact point and final detection point of the finger (1) based on pressure changes, and to identify the direction of movement and distance of movement of the finger (1).

[0117] For example, the artificial intelligence model (1000) can identify that the user's finger (1) is continuously pressing a specific area among a plurality of areas (10) included in the button (210) when the pressure between the first threshold and the second threshold is maintained in a specific area for a certain period of time, and can identify the directionality of the user gesture when the area where pressure is detected moves in the first direction or the second direction in this state.

[0118] For example, the artificial intelligence model (1000) can identify the point where contact of the finger (1) started and ended among a plurality of areas (10) included in the button (210) according to user input of swiping the button (210), based on detection data acquired by each of the first pressure sensor (170a) and the second pressure sensor (170b) at preset time intervals.

[0119] According to an embodiment, the electronic device (100) can identify, according to a predefined motion rule or an artificial intelligence model (1000), the movement of a position where pressure is detected between a first threshold value and a second threshold value, and a user gesture (e.g., a swipe gesture in the first direction, a swipe gesture in the second direction) according to the direction in which the position where pressure is detected has moved.

[0120] For example, the electronic device (100) may receive user input by a swipe gesture in which the user's finger (1) slides a certain distance while in contact with the button (210). For example, the swipe gesture may include a first directional swipe gesture in which the user's finger (1) swipes from the top part of the button (210) (e.g., first pressing part (213a)) toward the bottom part of the button (210) (e.g., second pressing part (213b)) along the length direction of the button (210), and a second directional swipe gesture in which the user's finger (1) swipes from the bottom part of the button (210) toward the top part of the button (210) along the length direction of the button (210).

[0121] For example, when a user's finger (1) comes into contact with any of the areas included in the first pressing part (213a), the first pressure sensor (170a) corresponding to the first pressing part (213a) detects pressure, but since a pressure below the first threshold is detected, it may not perform an action corresponding to a pressing gesture. When the position of the user's finger (1) moves on the button (210), the intensity of the pressure detected by the first pressure sensor (170a) and the intensity of the pressure detected by the second pressure sensor (170b) may change.

[0122] According to an embodiment, the electronic device (100) can identify the change in pressure intensity detected by the first pressure sensor (170a) and the change in pressure intensity detected by the second pressure sensor (170b) based on the detection data of a time series acquired by each of the first pressure sensor (170a) and the second pressure sensor (170b) at preset time intervals.

[0123] According to an embodiment, the electronic device (100) can identify a first direction swipe gesture or a second direction swipe gesture based on a change in pressure intensity detected by a first pressure sensor (170a) and a change in pressure intensity detected by a second pressure sensor (170b).

[0124] According to an embodiment, the electronic device (100) can perform an action corresponding to a swipe gesture in a first direction or a swipe gesture in a second direction.

[0125] For example, the electronic device (100) can perform actions such as scrolling down the screen of the display (140) based on a first direction swipe gesture, zooming out the screen, minimizing the app displayed on the screen, moving the cursor or selection pointer displayed on the screen down, and highlighting the bottom menu among the menus.

[0126] For example, the electronic device (100) can perform actions such as scrolling the screen of the display (140) upward based on a second swipe gesture, zooming in on the screen, maximizing a minimized app to display it on the screen, moving a cursor or selection pointer displayed on the screen upward, and highlighting the top menu among the menus.

[0127] According to an embodiment, an artificial intelligence model (1000) can predict (or identify) user gestures based on various training data and perform actions corresponding to user gestures (e.g., quick search, menu navigation).

[0128] According to an embodiment, the artificial intelligence model (1000) can adjust the detection sensitivity according to the speed of movement or the intensity of pressure of the user's finger (1). For example, if the finger (1) moves quickly, the detection sensitivity is increased to identify swipe gestures in real time based on time-series detection data, and if the finger (1) moves slowly, the detection sensitivity is lowered to identify swipe gestures based on the movement of the finger (1) on the button (210) with higher accuracy. For example, if the finger (1) moves slowly, the artificial intelligence model (1000) lowers the detection sensitivity to identify the distance and direction of movement of the finger (1) with relatively high accuracy.

[0129] According to an embodiment, when a swipe gesture is identified, the electronic device (100) may provide vibration (e.g., haptic feedback) or visual feedback to indicate that a swipe gesture has been identified. A specific description thereof will be provided later with reference to FIGS. 10 and FIGS. 11.

[0130] According to an embodiment, a predefined operation rule or artificial intelligence model (1000) can be created through learning.

[0131] Here, being created through learning means that a predefined operation rule or artificial intelligence model (1000) of the desired characteristics is created by applying a learning algorithm to a number of learning data. This learning may be performed on the electronic device (100) itself where the artificial intelligence according to the present disclosure is performed, or it may be performed through a separate server / system.

[0132] An artificial intelligence model (1000) may be composed of multiple neural network layers. At least one layer has at least one weight value and performs the layer's operation through the result of the operation of the previous layer and at least one defined operation. Examples of neural networks include CNN (Convolutional Neural Network), DNN (Deep Neural Network), RNN (Recurrent Neural Network), RBM (Restricted Boltzmann Machine), DBN (Deep Belief Network), BRDNN (Bidirectional Recurrent Deep Neural Network), Deep Q-Networks, and Transformers, and the neural networks in this disclosure are not limited to the aforementioned examples except where specified.

[0133] A learning algorithm is a method of training a specific target device (e.g., a robot) using multiple learning data to enable the specific target device to make decisions or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, and the learning algorithms in this disclosure are not limited to the aforementioned examples except where specified. A more detailed description of the artificial intelligence model (1000) will be provided with reference to FIG. 9.

[0134]

[0135] FIG. 9 is a diagram illustrating training data of an artificial intelligence model according to an embodiment of the present disclosure.

[0136] The electronic device (100) according to the present disclosure can recognize user gestures using an artificial intelligence model (1000). The artificial intelligence model can be trained to accurately identify the user's intention by comprehensively learning the pressure changes and movement paths that occur when the user swipes the button (210). For example, since the size or pressure of the finger (1) may differ for each user, the electronic device (100) can collect various learning data to train the artificial intelligence model (1000) and provide an artificial intelligence model optimized for the user of the electronic device (100).

[0137] Referring to FIG. 9, when a user's finger (1) contacts any one of the areas corresponding to the first pressing part (213a) (e.g., first area (10-1) to fourth area (10-4)) and the first pressure sensor (170a) detects pressure, the intensity of the pressure detected by the first pressure sensor (170a) may differ depending on the area contacted by the user's finger (1) or depending on the user. For example, the intensity when the finger (1) contacts the first area (10-1) among the first area (10-1) to fourth area (10-4) and the pressure between the first threshold value and the second threshold value is detected may be different from the intensity when the finger (1) contacts the fourth area (10-4) and the pressure between the first threshold value and the second threshold value is detected.

[0138] According to an embodiment, the electronic device (100) can acquire learning data based on the intensity of pressure when a user's first-direction swipe gesture to a button (210) according to all possible cases is input.

[0139] For example, referring to FIG. 9, the electronic device (100) can train an artificial intelligence model (1000) using detection data acquired by a first pressure sensor (170a) at a preset time interval and detection data acquired by a second pressure sensor (170b) at a preset time interval when the location where pressure between a first threshold value and a second threshold value is detected moves from a first area (10-1) to a second area (10-2) or from a first area (10-1) to an eighth area (10-8).

[0140] For example, the electronic device (100) has i) a first directional swipe gesture with a first area (10-1) as the starting point and any one of the second area (10-2) to the eighth area (10-8) as the ending point, ii) a first directional swipe gesture with a second area (10-2) as the starting point and any one of the third area (10-3) to the eighth area (10-8) as the ending point, iii) a first directional swipe gesture with a third area (10-3) as the starting point and any one of the fourth area (10-4) to the eighth area (10-8) as the ending point, iv) a first directional swipe gesture with a fourth area (10-4) as the starting point and any one of the fifth area (10-5) to the eighth area (10-8) as the ending point, v) a fifth area (10-5) as the starting point and a sixth When a first direction swipe gesture with any one of the areas (10-6) to the eighth area (10-8) as the end point, vi) a first direction swipe gesture with the sixth area (10-6) as the start point and the seventh area (10-7) or the eighth area (10-8) as the end point, vii) a first direction swipe gesture with the seventh area (10-7) as the start point and the eighth area (10-8) as the end point is input, the first pressure sensor (170a) and the second pressure sensor (170b) each detect time series detection data at preset time intervals, and the artificial intelligence model (1000) can be trained using the time series detection data as training data.

[0141] According to an embodiment, the electronic device (100) can acquire learning data based on the intensity of pressure when a user's second-direction swipe gesture to a button (210) according to all possible cases is input.

[0142] For example, referring to FIG. 9, the electronic device (100) can train an artificial intelligence model (1000) using detection data acquired by a first pressure sensor (170a) at a preset time interval and detection data acquired by a second pressure sensor (170b) at a preset time interval when the location where pressure between a first threshold value and a second threshold value is detected moves from the eighth area (10-8) to the seventh area (10-7) or from the eighth area (10-8) to the seventh area (10-7).

[0143] For example, the electronic device (100) has i) a second-direction swipe gesture with the eighth region (10-8) as the starting point and any one of the first region (10-1) to the seventh region (10-7) as the ending point, ii) a second-direction swipe gesture with the seventh region (10-7) as the starting point and any one of the first region (10-1) to the sixth region (10-6) as the ending point, iii) a second-direction swipe gesture with the sixth region (10-6) as the starting point and any one of the first region (10-1) to the fifth region (10-8) as the ending point, iv) a second-direction swipe gesture with the fifth region (10-5) as the starting point and any one of the first region (10-1) to the fourth region (10-4) as the ending point, v) a fourth region (10-4) as the starting point and the first When a second directional swipe gesture with any one of the areas (10-1) to the third area (10-3) as the end point, vi) a second directional swipe gesture with the third area (10-3) as the start point and the first area (10-1) or the second area (10-2) as the end point, vii) a second directional swipe gesture with the second area (10-2) as the start point and the first area (10-1) as the end point is input, the artificial intelligence model (1000) can be trained using the time series detection data detected by each of the first pressure sensor (170a) and the second pressure sensor (170b) at preset time intervals as training data.

[0144] According to an embodiment, the electronic device (100) trains an artificial intelligence model (1000) to identify the point where contact of the finger (1) begins and the point where contact ends after the finger (1) moves on the button (210) based on the detection data of a time series acquired by the first pressure sensor (170a) and the second pressure sensor (170b) at preset time intervals, and trains the artificial intelligence model (1000) to identify the distance the finger (1) has moved on the button (210) based on the point where contact of the finger (1) begins and the point where contact ends.

[0145] According to an embodiment, the electronic device (100) can receive detection data from the first pressure sensor (170a) and the second pressure sensor (170b) at preset time intervals to obtain detection data in a time series.

[0146] For example, the electronic device (100) can obtain time-series detection data from the first pressure sensor (170a) and the second pressure sensor (170b) based on the Hardware Abstraction Layer (HAL) pass-through method.

[0147] According to an embodiment, the HAL may be a hardware abstraction layer that hides hardware details and provides them to the operating system through a standardized interface so that the operating system of the electronic device (100) and the hardware can interact. According to an embodiment, based on the HAL, the operating system (or software) may be able to interact with the hardware without relying on the specificity of the hardware.

[0148] According to an embodiment, the electronic device (100) can acquire time-series sensing data based on the HAL passthrough method. For example, the HAL passthrough method can transmit RAW input data directly to the upper layer of the HAL as input to an artificial intelligence model (e.g., machine learning model) without any processing, rather than performing the original purpose of the HAL.

[0149] According to an embodiment, the HAL passthrough method allows the operating system (or software) to acquire time-series sensing data without an intermediate layer (e.g., HAL) and directly control the first pressure sensor (170a) and the second pressure sensor (170b), thereby maximizing performance (e.g., latency is reduced and bottlenecks can be prevented).

[0150] According to an embodiment, the operating system of the electronic device (100) obtains time-series detection data from each of the first pressure sensor (170a) and the second pressure sensor (170b) without an intermediate layer, and can identify the movement path of the location where pressure is detected between the first threshold value and the second threshold value (e.g., the starting point and the ending point where pressure is detected).

[0151] Additionally, among a plurality of regions included in the button (210) (e.g., a first region (10-1) to an eighth region (10-8)), if the region corresponding to the current position of the finger (1) changes as the finger (1) moves on the button (210) (e.g., if the region corresponding to the current position of the finger (1) on the button (210) changes from the first region (10-1) to the second region (10-2), the electronic device (100) may generate a signal to provide haptic feedback indicating that the region has changed. This will be explained in detail with reference to FIGS. 10 and FIGS. 11.

[0152]

[0153] FIG. 10 is a drawing for explaining UI and haptic feedback according to user gestures according to an embodiment of the present disclosure.

[0154] The electronic device (100) can be controlled based on the press gesture for the first press part (213a) when the first pressure sensor (170a) detects a pressure greater than or equal to the first threshold value according to the user's press gesture for the first press part (213a).

[0155] For example, the electronic device (100) can provide a UI that increases volume and a UI that increases brightness adjacent to the button (210) on the screen.

[0156] According to an embodiment, the electronic device (100) can control the electronic device (100) based on the press gesture for the second press part (213b) when the second pressure sensor (170b) detects pressure greater than or equal to the second threshold value in accordance with the user's press gesture for the second press part (213b). For example, the electronic device (100) can provide a UI for reducing volume and a UI for reducing brightness adjacent to the button (210) on the screen.

[0157] According to an embodiment, the electronic device (100) can identify that a part of the user's body (e.g., finger (1)) has come into contact with the button (210) when pressure between a first threshold and a second threshold is detected through the first pressure sensor (170a) or the second pressure sensor (170b). According to an embodiment, the electronic device (100) can identify that a part of the user's body has come into contact with the button (210) for a certain period of time when pressure above the first threshold is not detected and pressure between the first threshold and a second threshold is detected for a certain period of time, and can identify the path along which the pressure-sensing location moves.

[0158] For example, the electronic device (100) receives time-series detection data from the first pressure sensor (170a) and the second pressure sensor (170b) at preset time intervals, inputs the time-series detection data into an artificial intelligence model (1000) to identify the path of movement of the location where pressure is detected, and can identify the user's swipe gesture based on this.

[0159] For example, the electronic device (100) can identify a first-direction swipe gesture in which a user's finger (1) in contact with the button (210) moves from the top to the bottom of the button (210), or a second-direction swipe gesture in which it moves from the bottom to the top.

[0160] According to an embodiment, the electronic device (100) can acquire a current control signal to provide haptic feedback when the area where the user's finger (1) is located on the button (210) changes from one of the multiple areas (10) to another adjacent area according to the user's swipe gesture using an artificial intelligence model (1000).

[0161] According to an embodiment, the electronic device (100) can control a vibration actuator provided in the electronic device (100) based on a current control signal. According to an embodiment, the vibration actuator may be substantially the same as the vibration actuator (135) described with reference to FIG. 1.

[0162] According to an embodiment, the vibration actuator (135) is included in the button module (200) and can provide haptic feedback through the first pressing part (213a) and the second pressing part (213b) included in the button (210).

[0163] According to an embodiment, the electronic device (100) can use an artificial intelligence model (1000) to provide haptic feedback to indicate that pressure caused by contact of the finger (1) is detected at a point different from the point where contact of the finger (1) began to be detected on the button (210) when the user's finger (1) moves from one of the multiple areas (10) (e.g., second area (10-2)) to another area (e.g., third area (10-3)).

[0164] For example, the electronic device (100) can control a current driver IC (referred to as 133 in FIG. 1) to supply power to a vibration actuator (135). The vibration actuator (135) vibrates according to the power pattern supplied by the current driver IC (133) to provide haptic feedback, and the user can quickly determine by touch (e.g., vibration) whether a swipe gesture has been detected as intended by the user.

[0165] For example, the electronic device (100) inputs time-series detection data into an artificial intelligence model (1000), and when pressure (e.g., pressure between a first threshold and a second threshold) is detected at a point moved 1 tick (e.g., distance from the center of one area to the center of another adjacent area) from the point where pressure was detected according to the user's swipe gesture, the electronic device (100) can control a vibration actuator (135) to provide haptic feedback through a button (210) for an extremely short time (e.g., several to tens of milliseconds).

[0166] As described above, the electronic device (100) can provide visual, tactile, and auditory feedback to the user after recognizing the user's gesture. For example, when the user swipes the button (210), an animation effect can be applied to the display (140) to indicate that the swipe motion has been recognized. Additionally, a fine vibration can be provided through the haptic driver IC (133) to tactilely convey to the user that the swipe motion has been recognized. By providing visual, tactile, and auditory feedback, the user can be helped to interact with the electronic device more intuitively.

[0167]

[0168] FIG. 11 is a drawing for explaining a UI according to a user gesture according to an embodiment of the present disclosure.

[0169] FIG. 11 is a drawing for explaining an embodiment in which an electronic device (100) detects a first-direction swipe gesture or a second-direction swipe gesture through a button (210) and performs a corresponding operation to provide various UIs.

[0170] Referring to FIG. 11, the electronic device (100) provides a circular menu UI, and the menu UI may include a plurality of selectable menu items.

[0171] According to an embodiment, when a swipe gesture in a first direction is detected, the electronic device (100) can sequentially select (or highlight) each of a plurality of menu items in a clockwise direction.

[0172] For example, an electronic device (100) detects a first-direction swipe gesture through an artificial intelligence model (1000), and if the distance a finger (1) moves on a button (210) according to the first-direction swipe gesture corresponds to 1 tick in the first direction, a menu item adjacent to the currently selected menu item in a clockwise direction can be selected (or highlighted).

[0173] According to an embodiment, when a swipe gesture in the second direction is detected, the electronic device (100) can sequentially select each of a plurality of menu items in a counterclockwise direction.

[0174] For example, the electronic device (100) detects a second-direction swipe gesture through an artificial intelligence model (1000), and if the distance the finger (1) moves on the button (210) according to the second-direction swipe gesture corresponds to 1 tick in the second direction, a menu item adjacent to the currently selected menu item in a counterclockwise direction can be selected.

[0175] According to an embodiment, the electronic device (100) may provide a password input UI in the form of a circular dial.

[0176] For example, the electronic device (100) selects at least one character among a plurality of characters (or numbers) included in the password input UI through a swipe gesture identified by an artificial intelligence model (1000), and if the selected at least one character corresponds to a password for unlocking, the electronic device (100) can be unlocked.

[0177] According to an embodiment, the electronic device (100) may provide a list-type menu UI, and the menu UI may include multiple menu items. According to an embodiment, the electronic device (100) may activate (or select, highlight) any one of the multiple menu items through a swipe gesture identified by an artificial intelligence model (1000).

[0178] According to various embodiments of the present disclosure, the electronic device (100) can identify a swipe gesture for the button (210) (e.g., a swipe gesture in the first direction, a swipe gesture in the second direction) with high accuracy in addition to a press gesture for the first pressing part (213a) or the second pressing part (213b), and can control the electronic device (100) using the swipe gesture.

[0179]

[0180]

[0181] *According to an embodiment, an electronic device (e.g., the electronic device (100) of FIG. 1) comprises a housing, a button provided on the side of the housing, a memory including one or more storage media for storing instructions, a printed circuit board including a first pressure sensor for detecting pressure applied through a first pressing portion of a button provided on the side of the housing of the electronic device and a second pressure sensor for detecting pressure applied through a second pressing portion of the button, and at least one processor including a processing circuit, wherein when the instructions are executed individually or collectively by the at least one processor, the electronic device performs an operation corresponding to the first pressing portion when a pressure greater than or equal to a first threshold value is detected at the first pressure sensor, performs an operation corresponding to the second pressing portion when a pressure greater than or equal to the first threshold value is detected at the second pressure sensor, and when a pressure between the first threshold value and the second threshold value is detected at the first pressure sensor or the second pressure sensor, a preset from the first pressure sensor and the second pressure sensor It can be configured to receive detection data at time intervals, input the detection data into an artificial intelligence model to identify a user gesture based on the movement of the position where the pressure is detected on the button, and perform an action corresponding to the user gesture.

[0182] According to an embodiment, the instructions allow the electronic device to divide the button into a plurality of regions, and when the pressure between the first threshold value and the second threshold value is detected in the first region among the plurality of regions through the first pressure sensor, or when the pressure is detected in the second region among the plurality of regions, the detected data received from the first pressure sensor and the second pressure sensor is input into the artificial intelligence model to identify the user gesture.

[0183] According to an embodiment, the artificial intelligence model can be trained to identify a swipe gesture in the first direction as the user gesture when the movement of the position corresponds to the first direction based on the detection data, and to identify a swipe gesture in the second direction as the user gesture when the movement of the position corresponds to the second direction.

[0184] According to an embodiment, the instructions may be configured such that the electronic device inputs the sensing data into the artificial intelligence model so that when the pressure between the first threshold value and the second threshold value is detected in the first area among the plurality of areas, and the position where the pressure is detected moves along the first direction, and then when the pressure less than the second threshold value is detected in the second area among the plurality of areas, the swipe gesture in the first direction is identified as the user gesture.

[0185] According to an embodiment, the instructions may be configured such that the electronic device inputs the sensing data into the artificial intelligence model so that the pressure between the first threshold value and the second threshold value is detected in the second area among the plurality of areas, and after the position where the pressure is detected moves along the second direction, if the pressure less than the second threshold value is detected in the first area among the plurality of areas, the swipe gesture in the second direction is identified as the user gesture.

[0186] According to an embodiment, the first region may be any one of the regions corresponding to the first pressing part among the plurality of regions, and the second region may be any one of the regions corresponding to the second pressing part among the plurality of regions.

[0187] According to an embodiment, the instructions may be configured to allow the electronic device to input the sensing data into the artificial intelligence model and, when the location where the pressure is detected is identified as having moved from the first region to the second region, to provide haptic feedback.

[0188] According to an embodiment, the artificial intelligence model is a model that identifies a user gesture based on detection data received at the preset time interval and generates a current control signal to provide haptic feedback according to the positional movement according to the user gesture, and the instructions may be configured to cause the electronic device to supply current to a vibration actuator equipped in the electronic device based on the current control signal to provide haptic feedback.

[0189] According to an embodiment, the sensing data received from the first pressure sensor and the second pressure sensor is time-series data, and the instructions may be configured to allow the electronic device to receive the sensing data from the first pressure sensor and the pressure sensor according to a Hardware Abstraction Layer (HAL) pass-through method.

[0190] According to an embodiment, the instructions allow the electronic device to identify a press gesture for the first pressing part and control the electronic device based on the press gesture for the first pressing part when a pressure greater than or equal to the first threshold value is detected at the first pressure sensor, and to identify a press gesture for the second pressing part and control the electronic device based on the press gesture for the second pressing part when a pressure greater than or equal to the first threshold value is detected at the second pressure sensor.

[0191]

[0192] According to an embodiment, a method of operation of an electronic device including a first pressing part and a second pressing part and a button provided on the side may include: an operation of performing an operation corresponding to the first pressing part when a pressure greater than or equal to a first threshold value is detected by a first pressure sensor that detects pressure applied through the first pressing part; an operation of performing an operation corresponding to the second pressing part when a pressure greater than or equal to the first threshold value is detected by a second pressure sensor that detects pressure applied through the second pressing part; an operation of receiving detection data from the first pressure sensor and the second pressure sensor at a preset time interval when a pressure between the first threshold value and the second threshold value is detected by the first pressure sensor or the second pressure sensor; an operation of inputting the detection data into an artificial intelligence model to identify a user gesture based on the movement of the position where the pressure is detected on the button; and an operation of performing an operation corresponding to the user gesture.

[0193] According to an embodiment, the operation of receiving the detection data may include dividing the button into a plurality of regions, and when the pressure between the first threshold value and the second threshold value is detected in the first region among the plurality of regions through the first pressure sensor, or when the pressure is detected in the second region among the plurality of regions, receiving the detection data from the first pressure sensor and the second pressure sensor at the preset time interval.

[0194] According to an embodiment, the artificial intelligence model can be trained to identify a swipe gesture in the first direction as the user gesture when the movement of the position corresponds to the first direction based on the detection data, and to identify a swipe gesture in the second direction as the user gesture when the movement of the position corresponds to the second direction.

[0195] According to an embodiment, the operation of identifying the user gesture may include inputting the detection data into the artificial intelligence model so that the pressure between the first threshold value and the second threshold value is detected in the first area among the plurality of areas, and after the position where the pressure is detected moves according to the first direction, if the pressure less than the second threshold value is detected in the second area among the plurality of areas, the operation of identifying the swipe gesture in the first direction as the user gesture.

[0196] According to an embodiment, the operation of identifying the user gesture may include inputting the detection data into the artificial intelligence model so that the pressure between the first threshold value and the second threshold value is detected in the second area among the plurality of areas, and after the position where the pressure is detected moves according to the second direction, if the pressure less than the second threshold value is detected in the first area among the plurality of areas, the operation of identifying the swipe gesture in the second direction as the user gesture.

[0197] According to an embodiment, the first region may be any one of the regions corresponding to the first pressing part among the plurality of regions, and the second region may be any one of the regions corresponding to the second pressing part among the plurality of regions.

[0198] According to an embodiment, the operation method may further include an operation of providing haptic feedback when the sensing data is input into the artificial intelligence model and the location where the pressure is detected is identified as having moved from the first region to the second region.

[0199] According to an embodiment, the artificial intelligence model is a model that identifies a user gesture based on detection data received at a preset time interval and generates a current control signal to provide haptic feedback according to a positional movement according to the user gesture, and the operation of providing haptic feedback may include supplying current to a vibration actuator equipped in the electronic device based on the current control signal to provide haptic feedback.

[0200] According to an embodiment, the detection data received from the first pressure sensor and the second pressure sensor is time-series data, and the operation of receiving the detection data at the preset time interval may include the operation of receiving the detection data from the first pressure sensor and the pressure sensor according to a Hardware Abstraction Layer (HAL) pass-through method.

[0201] According to an embodiment, an operation for performing an operation corresponding to the first pressing part includes, when a pressure greater than or equal to the first threshold value is detected by the first pressure sensor, identifying a press gesture for the first pressing part and controlling the electronic device based on the press gesture for the first pressing part, and an operation for performing an operation corresponding to the second pressing part may include, when a pressure greater than or equal to the first threshold value is detected by the second pressure sensor, identifying a press gesture for the second pressing part and controlling the electronic device based on the press gesture for the second pressing part.

[0202]

[0203] However, various embodiments of the present disclosure may be applied to various types of electronic devices.

[0204] Meanwhile, the various embodiments described above may be implemented in a recording medium readable by a computer or a similar device using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented as the processor itself. According to software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

[0205] Meanwhile, computer instructions for performing processing operations of an electronic device according to the various embodiments of the present disclosure described above may be stored in a non-transitory computer-readable medium. When computer instructions stored in such a non-transitory computer-readable medium are executed by a processor of a specific device, they cause the specific device to perform processing operations in the electronic device according to the various embodiments described above.

[0206] A non-transient computer-readable medium refers to a medium that stores data semi-permanently and can be read by a device, unlike media that store data for a short period of time such as registers, caches, and memory. Specific examples of non-transient computer-readable media include CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, and ROMs.

[0207] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

1. In an electronic device, Housing; A button provided on the side of the above housing; Memory that stores instructions and includes one or more storage media; A printed circuit board comprising a first pressure sensor for detecting pressure applied through a first pressing portion of a button provided on the side of the housing of the electronic device and a second pressure sensor for detecting pressure applied through a second pressing portion of the button; and at least one processor including a processing circuit; and When the above instructions are executed individually or collectively by the at least one processor, the electronic device, When a pressure greater than or equal to a first threshold value is detected by the first pressure sensor, an operation corresponding to the first pressing part is performed, and When a pressure greater than or equal to the first threshold value is detected by the second pressure sensor, an operation corresponding to the second pressing part is performed, and When a pressure between the first threshold value and the second threshold value is detected at the first pressure sensor or the second pressure sensor, detection data is received from the first pressure sensor and the second pressure sensor at preset time intervals, and The above detection data is input into an artificial intelligence model to identify a user gesture based on the movement of the position where the pressure is detected on the button, and An electronic device configured to perform an action corresponding to the above user gesture.

2. In Paragraph 1, The above instructions cause the electronic device, An electronic device that divides the above button into a plurality of regions, and when the pressure between the first threshold value and the second threshold value is detected in the first region among the plurality of regions through the first pressure sensor, or when the pressure is detected in the second region among the plurality of regions, inputs the detection data received from the first pressure sensor and the second pressure sensor into the artificial intelligence model to identify the user gesture.

3. In Paragraph 2, The above artificial intelligence model is, Based on the above detection data, if the movement of the above position corresponds to the first direction, the swipe gesture of the first direction is identified as the user gesture, and An electronic device learned to identify a swipe gesture in the second direction as the user gesture when the movement of the above position corresponds to the second direction.

4. In Paragraph 3, The above instructions cause the electronic device, An electronic device configured to input the above detection data into the above artificial intelligence model so that the pressure between the first threshold value and the second threshold value is detected in the first area among the plurality of areas, and after the position where the pressure is detected moves along the first direction, if the pressure less than the second threshold value is detected in the second area among the plurality of areas, the swipe gesture in the first direction is identified as the user gesture.

5. In Paragraph 3, The above instructions cause the electronic device, An electronic device configured to input the above detection data into the above artificial intelligence model so that the pressure between the first threshold value and the second threshold value is detected in the second region among the plurality of regions, and after the location where the pressure is detected moves along the second direction, if the pressure less than the second threshold value is detected in the first region among the plurality of regions, the swipe gesture in the second direction is identified as the user gesture.

6. In Paragraph 2, The above-mentioned first region is, It is any one of the regions corresponding to the first pressing part among the plurality of regions above, and The above second region is, An electronic device that is one of the regions corresponding to the second pressing part among the plurality of regions above.

7. In Paragraph 2, The above instructions cause the electronic device, An electronic device configured to provide haptic feedback when the above detection data is input into the artificial intelligence model and the location where the pressure is detected is identified as having moved from the first region to the second region.

8. In Paragraph 7, The above artificial intelligence model is, A model that identifies a user gesture based on detection data received at the above-mentioned preset time intervals and generates a current control signal to provide haptic feedback according to positional movement corresponding to the user gesture. The above instructions cause the electronic device, An electronic device configured to provide haptic feedback by supplying current to a vibration actuator equipped in the electronic device based on the above current control signal.

9. In Paragraph 1 or 2, The detection data received from the first pressure sensor and the second pressure sensor is time-series data, and The above instructions cause the electronic device, An electronic device configured to receive the detection data from the first pressure sensor and the pressure sensor according to a HAL (Hardware Abstraction Layer) pass-through method.

10. In Paragraph 1 or 2, The above instructions cause the electronic device, When a pressure greater than or equal to the first threshold value is detected at the first pressure sensor, a press gesture for the first pressing part is identified, and the electronic device is controlled based on the press gesture for the first pressing part. An electronic device configured to identify a press gesture for the second pressing part and control the electronic device based on the press gesture for the second pressing part when a pressure greater than or equal to the first threshold value is detected at the second pressure sensor.

11. A method of operation of an electronic device comprising a first pressing portion and a second pressing portion, and a button provided on a side, wherein An operation to perform an operation corresponding to the first pressing part when a pressure greater than or equal to a first threshold value is detected by a first pressure sensor that detects pressure applied through the first pressing part; When a pressure greater than or equal to the first threshold value is detected by a second pressure sensor that detects pressure received through the second pressing part, an operation corresponding to the second pressing part is performed; When a pressure between the first threshold value and the second threshold value is detected at the first pressure sensor or the second pressure sensor, the operation of receiving detection data from the first pressure sensor and the second pressure sensor at a preset time interval; An operation of inputting the above detection data into an artificial intelligence model to identify a user gesture based on the movement of the position where the pressure is detected on the button; and A method of operation comprising: an action that performs an action corresponding to the above user gesture.

12. In Paragraph 11, The operation of receiving the above detection data is, A method of operation comprising: dividing the above button into a plurality of regions, and when the pressure between the first threshold value and the second threshold value is detected in the first region among the plurality of regions through the first pressure sensor, or when the pressure is detected in the second region among the plurality of regions, receiving the detection data from the first pressure sensor and the second pressure sensor at the preset time interval; and 13. In Paragraph 12, The above artificial intelligence model is, Based on the above detection data, if the movement of the above position corresponds to the first direction, the swipe gesture of the first direction is identified as the user gesture, and A method of operation learned to identify a swipe gesture in the second direction as the user gesture when the movement of the above position corresponds to the second direction.

14. In Paragraph 13, The action of identifying the above user gesture is, A method of operation comprising: inputting the above detection data into the artificial intelligence model, wherein the pressure between the first threshold value and the second threshold value is detected in the first region among the plurality of regions, and after the position where the pressure is detected moves according to the first direction, the pressure less than the second threshold value is detected in the second region among the plurality of regions, and identifying the swipe gesture in the first direction as the user gesture.

15. In a non-transient storage medium for storing computer-readable instructions, said instructions, when executed by at least one processor of an electronic device comprising a housing and a button provided on the side of the housing, said electronic device, When a pressure greater than a first threshold value is detected by a first pressure sensor that detects pressure applied through a first pressing portion of a button provided on the side of the housing, an operation corresponding to the first pressing portion is performed, When a pressure greater than or equal to the first threshold value is detected by a second pressure sensor that detects pressure applied through the second pressing part of the button, an operation corresponding to the second pressing part is performed, When a pressure between the first threshold value and the second threshold value is detected at the first pressure sensor or the second pressure sensor, detection data is received from the first pressure sensor and the second pressure sensor at preset time intervals, and The above detection data is input into an artificial intelligence model to identify a user gesture based on the movement of the position where the pressure is detected on the button, and A storage medium that causes an action corresponding to the above user gesture to be performed.

Images