Gesture control method and system, and related apparatus

Abstract

The present application relates to the technical field of electronics, and discloses a gesture control method and system, and a related apparatus. The method is applied to a first electronic device, and a plurality of actions are pre-stored in the first electronic device. The method comprises: detecting a first action of a user; detecting a second action of the user; detecting that the user ends a gesture; and executing a first operation corresponding to the first gesture, wherein the first gesture comprises the first action and the second action, and the start action of the first gesture is the first action. In this way, gesture recognition can be implemented on the basis of action recognition, and an infinite number of gestures can be constructed on the basis of a finite number of actions, thereby meeting various functional requirements of a user in different scenarios.

Description

A gesture control method, system and related device

[0001] This application claims priority to Chinese Patent Application No. 202411545197.7, filed on October 31, 2024, entitled “A gesture control method, system and related device”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of electronic technology, and in particular to a gesture control method, system and related device. Background Technology

[0003] With the continuous development of electronic technology, electronic devices are becoming increasingly diversified, and more and more electronic devices are equipped with gesture recognition capabilities. Electronic devices can pre-store one or more gestures, the characteristics of each gesture, and the corresponding operation for each gesture. When a user's hand movement matches the characteristics of any pre-stored gesture, the electronic device can recognize that the user is performing the gesture corresponding to that characteristic and execute the operation corresponding to that gesture.

[0004] However, electronic devices can only recognize a limited number of gestures, making it difficult to match diverse functions. Summary of the Invention

[0005] This application provides a gesture control method, system, and related device, which realizes gesture recognition based on action recognition, and can trigger electronic devices to control themselves or other electronic devices to perform specified operations through gestures.

[0006] In a first aspect, this application provides a gesture control method applied to a first electronic device, the first electronic device having multiple pre-stored actions; the method includes: detecting a first action of a user; detecting a second action of a user; and in response to the user ending the first gesture, performing a first operation corresponding to the first gesture, the first gesture including a first action and a second action, and the starting action of the first gesture being the first action.

[0007] In this application, a gesture can include one or more actions; that is, an action is used as the basic unit, and a complete gesture is constructed based on the actions. In this way, gesture recognition can be achieved based on action recognition, and an infinite number of gestures can be constructed from a finite number of actions, meeting the diverse functional needs of users in different scenarios. Furthermore, since a gesture can consist of many actions, action-based gesture recognition, compared to direct gesture recognition, has more lenient time requirements for user gesture execution. Users can perform gestures more readily without worrying about gesture recognition failure due to slow execution.

[0008] In one possible implementation, the ending action of the first gesture is the second action; in response to the user ending the first gesture, specifically including: no user action is detected within a first time period, the start time of the first time period is the time when the second action is detected, and the duration of the first time period is a first duration.

[0009] In this way, after an action is performed, as long as the user performs the next action within a preset time (e.g., 500 milliseconds, 300 milliseconds, etc.), it can be determined that the two actions performed by the user belong to the same gesture, thus realizing the continuity detection of gestures.

[0010] In one possible implementation, responding to the user ending the first gesture specifically includes: determining that among the one or more gestures stored in the first electronic device, there is no gesture that includes a first action and a second action, whose starting action is the first action, and whose number of actions is greater than the number of actions detected.

[0011] In this way, the electronic device can determine that the user's gesture has ended when there is no gesture in the pre-stored gestures that has more actions than the currently executed gesture, and the action sequence composed of the previous one or more actions is completely consistent with the currently executed gesture.

[0012] In one possible implementation, after detecting that the user has performed a first action, the method further includes: outputting a first prompt, the first prompt including one or more action guides, each action guide including one or more actions, the one or more actions being actions that the user can perform after the first action.

[0013] In this way, during the user's gesture execution, a first prompt can be output based on the sequence of actions the user has already performed, indicating the actions the user can perform next.

[0014] In one possible implementation, one or more action guides include a first action guide, which in turn includes a second action.

[0015] In this way, the first action can guide the user and prompt them to perform the second action after the first action.

[0016] In one possible implementation, one or more action guides may further include a third action guide, which includes a fourth action, and the third action guide is displayed after the first action guide, wherein the historical execution count of the fourth action is lower than the historical execution count of the second action.

[0017] In this way, the display order of one or more actions can be determined based on the historical number of times the actions have been executed.

[0018] In one possible implementation, one or more action guides may also include a fourth action guide, with the first action guide appearing before the fourth action guide, and the first action guide being more relevant to the first scene than the fourth action guide is to the first scene.

[0019] In this way, the display order of one or more action guides can be determined based on the relevance of the action guide to the scene. The relevance of the action guide to the scene can refer to the relevance of the gesture that the action guide instructs the user to perform to the scene.

[0020] In one possible implementation, the first action guidance also includes a name for the first operation, which indicates the operation triggered by the user after performing the first gesture.

[0021] In this way, the user can be guided by the first action and prompted to perform the second action after the first action, which can trigger the first electronic device to perform the first operation.

[0022] In one possible implementation, the first prompt also includes a first action string, which is used to indicate an action performed by the user, and the first action string includes a first action.

[0023] In this way, the user can also be prompted with the first action string in the first prompt, i.e. the first action.

[0024] In one possible implementation, the first gesture also includes a third action; after detecting that the user has performed the first action, the method further includes: detecting that the user has performed the third action.

[0025] Thus, the first gesture can include three or more actions.

[0026] In one possible implementation, outputting a first prompt specifically includes: displaying a first action string based on a first display method; and displaying one or more action guides based on a second display method, wherein the first display method and the second display method are different.

[0027] The first display mode can be used to indicate that the first action sequence is an executed action, and the second display mode can be used to indicate that the action guide is a subsequent action that can be executed. In one possible implementation, the first electronic device can distinguish between the first and second display modes by different display modes such as display color, font, font size, and weight. In another possible implementation, the first electronic device can also distinguish between the first and second display modes by different symbols.

[0028] In this way, different display methods can be used to prompt the user about the actions that have been performed and the actions that can be performed next.

[0029] In one possible implementation, after detecting the user's second action, the method further includes: outputting a second prompt, the second prompt being used to prompt the user that ending the gesture can trigger the first electronic device to perform the first operation.

[0030] In this way, when a user's gesture can trigger a specific operation, the user can be prompted to end the gesture in order to trigger that specific operation.

[0031] In one possible implementation, the first action guide further includes a third action, which is displayed after the second action; after detecting the user's second action, the method further includes: outputting a third prompt; the third prompt includes the second action guide, which includes the third action; and / or, the third prompt further includes a second action string, which indicates an action performed by the user, and the second action string includes the first action and the second action.

[0032] In this way, after the user performs the second action, a third prompt can be output. The third prompt can be used to inform the user of changes in the action that has been performed, and / or to inform the user of subsequent actions that can be performed.

[0033] In one possible implementation, before detecting the user's first action, the method further includes: establishing a first communication connection with a second electronic device; and performing a first operation corresponding to the first gesture, specifically including: sending a first instruction to the second electronic device through the first communication connection, the first instruction being used to instruct the second electronic device to perform a second operation.

[0034] In this way, a first gesture can trigger a first electronic device to control a second electronic device to perform a second operation.

[0035] In one possible implementation, the method further includes: obtaining a first device type of the second electronic device based on the first communication connection; and sending a first instruction to the second electronic device through the first communication connection, specifically including: sending a first instruction to the second electronic device based on the first device type through the first communication connection.

[0036] In this way, it is possible to determine whether the second operation is performed by controlling the second electronic device through the first electronic device, based on the device type of the second electronic device and the first gesture performed by the user.

[0037] In one possible implementation, before detecting the user's first action, the method further includes: displaying a first interface, the first interface including a first control; receiving and responding to the user's operation on the first control, displaying a second interface, the second interface including a first action option, the first action option corresponding to the first action; receiving and responding to the user's operation on the first action option, displaying one or more action options, the first or more action options including a second action option, the second action option corresponding to the second action; receiving and responding to the user's operation on the second action option, determining that the first gesture includes a first action and a second action, and that the first action is the starting action of the first gesture.

[0038] In this way, the first electronic device can create custom gestures based on the user's selected action sequence, making it easier for users to create gestures according to their own needs and enriching the diversity of gestures.

[0039] In one possible implementation, the second interface further includes a second action option and a third action option, the third action option corresponding to a fourth action, and there is no continuity between the fourth action and the first action; receiving and responding to the user's operation on the first action option, displaying one or more action options, specifically including: receiving and responding to the user's operation on the first action option, displaying the second action option, and not displaying the third action option.

[0040] In this way, during the user's gesture creation process, the first electronic device can display action options that are sequential with the last selected action. Since the fourth action corresponding to the third action option is not sequential with the first action, the first electronic device may not display the third action option after the user selects the first action.

[0041] In one possible implementation, after determining that the first gesture includes a first action and a second action, the method further includes: outputting a test prompt to prompt the user to perform the first gesture; detecting that the user has performed the first gesture; and outputting a success message to prompt the user that the first gesture has been successfully created.

[0042] In this way, after the gesture is created, the user can be prompted to test whether the gesture can be successfully recognized, thereby improving the success rate of subsequent gesture recognition.

[0043] In one possible implementation, outputting test prompts specifically includes: displaying a test interface, the test interface including a first identification identifier corresponding to the first action and a second identification identifier corresponding to the second action, both the first identification identifier and the second identification identifier being in a pending identification state; detecting that the user performs a first gesture specifically includes: detecting that the user performs a first action; the first identification identifier changing from a pending identification state to a successful identification state; detecting that the user performs a second action; the second identification identifier changing from a pending identification state to a successful identification state.

[0044] In this way, during the gesture testing process, the user can be prompted with the recognition status of each action in the gesture, so that the user can keep track of the gesture test status in real time.

[0045] In one possible implementation, before detecting that the user has performed the first gesture, the method further includes: after outputting a test prompt, detecting a test failure; outputting a test failure prompt, which is used to inform the user that the first gesture has failed the test; receiving and responding to the user's operation to retest the first gesture, displaying a test interface, in which both the first identification identifier and the second identification identifier are in a state of waiting to be identified.

[0046] This way, if the gesture test fails, you can restart the gesture test.

[0047] In one possible implementation, after outputting a test failure message, the system receives and responds to the user's operation to retest the first gesture, and displays the test interface. Specifically, this includes receiving and responding to the user's operation to retest the first gesture, and if the number of times the first gesture test fails is less than a preset threshold, displaying the test interface.

[0048] In this way, if a gesture test fails, it can be determined whether to retest the gesture based on the number of failures.

[0049] In one possible implementation, if the number of failures is greater than or equal to a preset threshold, the first electronic device may also prompt the user to re-edit the first gesture.

[0050] In one possible implementation, after determining the first gesture, the method further includes: displaying a third interface, the third interface including one or more operation options, the one or more operation options including the first operation option; receiving and responding to the user's operation on the first operation option, and determining the operation corresponding to the first gesture as the first operation.

[0051] This way, after creating the first gesture, you can set the operation corresponding to the first gesture as the first operation.

[0052] In one possible implementation, receiving and responding to the user's operation on the first operation option and determining the operation corresponding to the first gesture as the first operation specifically includes: receiving the user's operation on the first operation option; if the security level of the first operation is lower than or equal to the security level of the first gesture, determining the operation corresponding to the first gesture as the first operation.

[0053] In this way, when setting the correspondence between the first gesture and the first operation, it can be determined whether the security level of the first operation is higher than that of the first gesture. If the security level of the first operation is lower than or equal to that of the first gesture, the operation corresponding to the first gesture can be set as the first operation. This can avoid the mismatch between the security of the operation and the gesture and improve the security of the operation triggering conditions.

[0054] In one possible implementation, the first electronic device stores a second gesture; the method further includes: receiving an operation from a user to set a correspondence between the second gesture and a third operation; if the security level of the third operation is higher than the security level of the second gesture, outputting a security prompt, the security prompt being used to indicate to the user that the security level of the third operation is higher than the security level of the second gesture.

[0055] In this way, the first electronic device can establish a correspondence between the second gesture and the third operation. If the security level of the second gesture is lower than that of the third operation, a security prompt can be output to remind the user that the security level of the third operation is higher than that of the second gesture, so that the user can reselect the gesture corresponding to the third operation or reselect the operation corresponding to the second gesture.

[0056] In one possible implementation, after outputting the security prompt, the method further includes: receiving an operation from the user to change a third operation to a fourth operation; if the security level of the fourth operation is lower than or equal to the security level of the second gesture, determining that the operation corresponding to the second gesture is the fourth operation; or, the electronic device also stores a third gesture; receiving an operation from the user to change a second gesture to a third gesture; if the security level of the third operation is lower than or equal to the security level of the third gesture, determining that the gesture corresponding to the third operation is the third gesture; or, receiving an operation from the user to edit a second gesture, determining the updated second gesture; if the security level of the third operation is lower than or equal to the security level of the updated second gesture, determining that the gesture corresponding to the third operation is the updated second gesture.

[0057] Thus, if the security level of the third operation is higher than that of the second gesture, the first electronic device can change the operation corresponding to the second gesture, or it can change the gesture corresponding to the third operation, or it can re-edit the second gesture.

[0058] Secondly, this application provides an electronic device, which includes one or more processors and one or more memories; wherein the one or more memories are coupled to one or more processors, and the one or more memories are used to store computer instructions, and when the one or more processors execute the computer instructions, they implement the gesture control method in any possible implementation of any of the above aspects.

[0059] Thirdly, this application provides a chip system comprising: a processing circuit and an interface circuit, wherein the interface circuit is used to receive code instructions and transmit them to the processing circuit, and the processing circuit is used to execute the code instructions to perform the gesture control method in any possible implementation of any of the above aspects.

[0060] Fourthly, this application provides a readable storage medium storing computer instructions that, when executed by a processor, implement the gesture control method in any of the possible implementations of any of the above aspects.

[0061] Fifthly, this application provides a computer program product including computer instructions, which, when executed by a processor, implements the gesture control method in any of the possible implementations of any of the above aspects.

[0062] The beneficial effects of aspects two through five can be referenced from the beneficial effects of aspect one above. Attached Figure Description

[0063] Figure 1A is a schematic diagram of the system architecture of a gesture control system provided in an embodiment of this application;

[0064] Figure 1B is a schematic diagram of the device configuration of an electronic device provided in an embodiment of this application;

[0065] Figure 1C is a schematic diagram of the device configuration of another electronic device provided in an embodiment of this application;

[0066] Figure 2 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0067] Figure 3 is a flowchart illustrating a gesture control method provided in an embodiment of this application;

[0068] Figure 4 is a flowchart illustrating a process for determining whether a user's gesture has been completed, according to an embodiment of this application.

[0069] Figure 5 is a schematic diagram of the process of an electronic device 200 performing operations based on a recognized gesture, according to an embodiment of this application.

[0070] Figure 6A is a schematic diagram of a gesture creation process provided in an embodiment of this application;

[0071] Figure 6B is a schematic diagram of the process of determining gesture 3 based on multiple user operations on the gesture creation interface provided in an embodiment of this application.

[0072] Figure 7 is a flowchart illustrating an operation for setting a gesture according to an embodiment of this application;

[0073] Figure 8 is a flowchart illustrating a gesture setting operation according to an embodiment of this application;

[0074] Figure 9 is a schematic diagram of the process of setting up an electronic device with a security level that matches the operation and gestures according to an embodiment of this application;

[0075] Figures 10A-10D are schematic diagrams of a set of user-executed gestures provided in the embodiments of this application;

[0076] Figures 11A-11N are schematic diagrams of a set of interfaces for answering / hanging up calls via gestures, provided in an embodiment of this application.

[0077] Figures 12A-12E are schematic diagrams of a set of interfaces for updating gesture operation prompts based on user-performed actions, provided in an embodiment of this application.

[0078] Figures 13A-13M are a set of schematic diagrams of interfaces for creating custom gestures provided in the embodiments of this application;

[0079] Figures 14A-14D are schematic diagrams of a set of interfaces for setting gestures according to embodiments of this application;

[0080] Figures 15A-15G are schematic diagrams of an interface showing a set of gestures that match the security level of the settings and operations provided in the embodiments of this application;

[0081] Figures 16A-16F are schematic diagrams of an interface provided in this application to increase the number of actions in a gesture when the gesture safety level is lower than the operation safety level;

[0082] Figures 16G-16K are schematic diagrams of an interface provided in this application embodiment, showing how to add high-security-level gestures when the gesture security level is lower than the operation security level.

[0083] Figures 17A-17D are schematic diagrams of an interface showing a set of gestures corresponding to selection operations provided in an embodiment of this application;

[0084] Figure 18 is a schematic diagram of the functional modules of an electronic device provided in an embodiment of this application;

[0085] Figure 19 is a schematic diagram of the functional modules of a gesture control system provided in an embodiment of this application;

[0086] Figure 20 is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application;

[0087] Figure 21 is a flowchart illustrating a gesture control method provided in an embodiment of this application. Detailed Implementation

[0088] The technical solutions in the embodiments of this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the word "and / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0089] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0090] The term "user interface (UI)" used in the following embodiments of this application refers to the medium interface through which an application or operating system interacts and exchanges information with the user. It realizes the conversion between the internal form of information and the form that the user can accept. The user interface is source code written in a specific computer language such as Java or Extensible Markup Language (XML). The interface source code is parsed and rendered on the electronic device, ultimately presenting content that the user can recognize. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be visible interface elements such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets displayed on the screen of an electronic device.

[0091] The following describes some terms used in the embodiments of this application.

[0092] Action: An action is composed of the movement of the fingers and / or arms. In the embodiments of this application, an action can be the smallest unit of action that an electronic device can recognize.

[0093] Gestures: Gestures are also called action sequences. An action sequence includes one or more actions performed in a specific order. When the execution order is different, the gestures (action sequences) performed by the user can be considered different. In the embodiments of this application, gestures can be used to trigger electronic devices to perform the operation corresponding to the gesture.

[0094] The following describes the system architecture of a gesture control system provided by an embodiment of this application.

[0095] Figure 1A is a schematic diagram of the system architecture of a gesture control system 10 provided in an embodiment of this application.

[0096] As shown in Figure 1A, the gesture control system 10 may include an electronic device 100. Optionally, the gesture control system 10 may also include an electronic device 200. A communication connection may be established between the electronic device 100 and the electronic device 200. This communication connection may be a wired communication connection or a wireless communication connection. The wireless communication connection may include a wireless communication connection established based on any of the following wireless communication technologies: wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), NearLink, and intrabody communication (IBC).

[0097] The electronic device 100 is equipped with a gesture recognition function, which can recognize the actions performed by the user and determine the gesture performed by the user based on the sequence of actions performed by the user. After recognizing the gesture performed by the user, the electronic device 100 can determine the corresponding operation based on the pre-stored correspondence between gestures and operations, and execute the operation.

[0098] In some embodiments, a gesture performed by a user can be used to control an electronic device 200 to perform a specified operation. In this case, when the electronic device 100 detects that the user has performed the gesture, it can send an instruction to the electronic device 200, which is used to instruct the electronic device 200 to perform the specified operation.

[0099] In some embodiments, electronic device 200 may receive instructions sent by electronic device 100 and execute the operation specified by the instructions.

[0100] In some embodiments, electronic device 100 may be a wearable device such as a watch (or bracelet, etc.) or earphones as shown in FIG. 1A, and electronic device 200 may be a device such as a smartphone. In other embodiments, electronic device 100 may also be other types of wearable devices (e.g., smart rings, smart armbands, smart glasses, etc.), and electronic device 200 may also be other types of electronic devices such as smart home devices, tablets, laptops, AR devices, and VR devices. This application does not limit the specific device type and device form of electronic device 100 and electronic device 200.

[0101] It is understood that the embodiment shown in Figure 1A is only an example. In the embodiments of this application, the gesture control system 10 may also include more, fewer, or different electronic devices than the above embodiments. For example, electronic device 100 can control multiple other electronic devices (such as electronic device 300, etc.) through gestures. This application does not limit this.

[0102] The following describes two device forms of the electronic device 100 provided in the embodiments of this application.

[0103] Figure 1B shows a schematic diagram of the device configuration of an electronic device 100 provided in an embodiment of this application.

[0104] As shown in Figure 1B, the electronic device 100 can be a watch 20. The watch 20 can include a watch body 21 and a watch strap 22. The watch body 21 can be equipped with a display screen, and optionally, it can also be equipped with one or more buttons (such as a crown 23). The watch body 21 of the watch 20 can also have a built-in motion recognition module, which can include any one or more of the following devices: accelerometer, gyroscope, touch sensor, pressure sensor, etc.

[0105] When a user wears watch 20, watch 20 can recognize one or more actions performed by the user through the aforementioned motion recognition module, determine the gesture performed by the user based on the recognized actions, and execute the operation corresponding to the gesture. During the user's gesture execution, watch 20 can display gesture operation prompts on the display screen based on the detected actions. These prompts can be used to indicate the currently available gestures and the corresponding operations for different gestures.

[0106] Figure 1C shows a schematic diagram of the device configuration of another electronic device 100 provided in an embodiment of this application.

[0107] As shown in Figure 1C, the electronic device 100 may be smart glasses 30, which may include a frame 31 and lenses 32. The smart glasses 30 may also include a motion recognition module. In some embodiments, the motion recognition module may include one or more cameras 33 disposed on the frame 31, which may be used to capture images (e.g., images of the user's hand).

[0108] When a user wears smart glasses 30, the smart glasses 30 can capture images of the user's hands through camera 33 and recognize the user's hand movements based on the captured images. Then, the smart glasses 30 can determine the gesture performed by the user based on one or more recognized movements and execute the corresponding operation.

[0109] It is understood that the embodiments shown in Figures 1B-1C above are only two examples. In the embodiments of this application, the electronic device 100 can also be a wearable device such as a wristband, smart ring, or smart armband. This application does not limit it here.

[0110] The hardware structure of the electronic device 100 provided in the embodiments of this application is described below.

[0111] Figure 2 shows a schematic diagram of the hardware structure of an electronic device 100 provided in an embodiment of this application.

[0112] The electronic device 100 can be a wearable device (also called a wearable device or wearable device) such as a watch, bracelet, smart glasses, or smart ring. This application embodiment does not impose any special restrictions on the specific type of the electronic device.

[0113] Electronic device 100 may include a processor 110, internal memory 121, charging management module 140, power management module 141, battery 142, wireless communication module 160, sensor module 180, display screen 194, etc. Optionally, electronic device 100 may also include any one or more of the following: external memory interface 120, universal serial bus (USB) interface 130, audio module 170, buttons 190, motor 191, indicator 192, photoplethysmography (PPG) module 195, etc.

[0114] The sensor module 180 may include a gyroscope sensor 180B, an accelerometer sensor 180E, and a touch sensor 180K. Optionally, the sensor module 180 may also include one or more of the following sensors: pressure sensor, barometric pressure sensor, magnetic sensor, distance sensor, proximity sensor, fingerprint sensor, temperature sensor, ambient light sensor, etc.

[0115] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0116] Processor 110 may include one or more processing units, such as application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0117] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0118] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0119] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0120] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0121] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0122] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, internal memory 121, display screen 194, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0123] The wireless communication module 160 can provide solutions for wireless communication applied to the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), NearLink, and intrabody communication (IBC). For example, when two electronic devices communicate using an IBC solution, both devices have at least one electrode that contacts the skin, through which they send and receive information via the human body. The wireless communication module 160 can be one or more devices integrating at least one communication processing module.

[0124] In some embodiments, the electronic device 100 may include an antenna, and the wireless communication module 160 may be coupled to the antenna, enabling the electronic device 100 to communicate with a network and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0125] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0126] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), or it can be manufactured using organic light-emitting diodes (OLEDs), active-matrix organic light-emitting diodes (AMOLEDs), flexible light-emitting diodes (FLEDs), minimized LEDs, microLEDs, micro-OLEDs, quantum dot light-emitting diodes (QLEDs), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0127] Internal memory 121 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM). The RAM can be directly read and written by the processor 110 and can be used to store executable programs (e.g., machine instructions) of the operating system or other running programs, as well as user and application data. The NVM can also store executable programs and user and application data, and can be pre-loaded into the RAM for direct read and write operations by the processor 110.

[0128] The external memory interface 120 can be used to connect to external non-volatile memory, thereby expanding the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to perform data storage functions. For example, music, video, and other files can be stored in the external non-volatile memory.

[0129] The audio module 170 may include one or more of the following: a speaker 170A, a receiver 170B, a microphone 170C, etc. The electronic device 100 can implement audio functions, such as music playback and recording, through the audio module 170 and an application processor.

[0130] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0131] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0132] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0133] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0134] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 about three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 180B.

[0135] The accelerometer 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, it can detect the magnitude and direction of gravity.

[0136] Touch sensor 180K, also known as a "touch device," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0137] In some embodiments, the electronic device 100 may also include one or more of the following sensors: temperature sensor, magnetic sensor, barometric pressure sensor, etc.

[0138] A temperature sensor is used to detect temperature. In some embodiments, the electronic device 100 can measure a user's body temperature using a temperature sensor. In other embodiments, the electronic device 100 can also measure the temperature of the user's environment using a temperature sensor.

[0139] Magnetic sensors can be used to detect ambient magnetic fields. In some embodiments, a magnetic sensor may include a Hall effect sensor.

[0140] Barometric pressure sensors can be used to detect the air pressure in a user's environment.

[0141] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0142] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can be corresponding to touch operations applied to different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations applied to different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0143] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0144] A PPG module can include a transmitter and a receiver. The transmitter can emit infrared or green light, and the receiver can receive infrared or green light reflected from biological tissues (such as skin, blood, etc.). The PPG module can measure PPG signals, which can be used to measure the user's physiological parameters, such as heart rate, blood oxygen saturation, and blood pressure.

[0145] This application provides a gesture control method. An electronic device 100 can pre-store multiple actions, and can also store one or more gestures and the correspondence between gestures and operations. Each gesture can be considered as an action sequence (including one or more actions executed in a specific order). The electronic device 100 can recognize one or more actions performed by the user. Based on the recognized one or more actions, the electronic device 100 can output gesture operation prompts. These prompts can be used to indicate subsequent executable actions (or action sequences) to the user, and optionally, can also be used to indicate the operation corresponding to the executable gesture. After determining that the user's gesture has been completed, the electronic device 100 can execute the operation corresponding to the gesture based on the direct correspondence between gestures and operations.

[0146] In this way, the electronic device 100 can recognize gestures (action sequences) by recognizing actions. Since different combinations of multiple actions can also form different gestures, the electronic device 100 can increase the diversity of gestures based on a limited number of actions, thereby reducing memory pressure.

[0147] The action can also be referred to as a key action frame or a gesture fragment. This application can identify gestures formed by combinations of different key action frames by recognizing the key action frames of a gesture, thereby expanding the number of gestures that the electronic device 100 can recognize.

[0148] The following describes the specific process of a gesture control method provided in this application.

[0149] Figure 3 shows a flowchart of a gesture control method provided in an embodiment of this application.

[0150] As shown in Figure 3, the specific process of a gesture control method may include the following steps:

[0151] S301. Electronic device 100 detects user-performed action 1.

[0152] Electronic device 100 may store an action library, which may include multiple actions and the characteristics of each action. For example, the actions stored in the action library may include, but are not limited to, any one or more of the following: clenching a fist, releasing, pinching, raising the wrist, keeping a fist clenched, swinging upwards, swinging downwards, sliding the thumb upwards on the side of the index finger, sliding the thumb downwards on the side of the index finger, tapping the thumb on the second joint side of the index finger, etc.

[0153] For example, Table 1 shows the composition of the action library stored in an electronic device 100 according to an embodiment of this application.

[0154] Table 1

[0155] As shown in Table 1, the action library stored in electronic device 100 can include multiple actions and features of each action. For example, the features of action P1 can include features A1 and A2; the features of action P2 can include features B1, B2, and B3; and the features of action P3 can include feature C1. That is, when electronic device 100 detects that the user's current action satisfies features A1 and A2, it can determine that the action performed by the user is action P1; when electronic device 100 detects that the user's current action satisfies features B1, B2, and B3, it can determine that the action performed by the user is action P2; and when electronic device 100 detects that the user's current action satisfies feature C1, it can determine that the action performed by the user is action P3.

[0156] It is understood that the embodiments shown in Table 1 are merely illustrative of the composition of the action library. In the embodiments of this application, the action library stored by the electronic device 100 may include more, fewer, or different actions than those shown in Table 1, and may also include more, fewer, or different features than those shown in the above embodiments, or may include a correspondence between actions and features that is different from those shown in the above embodiments. This application does not limit this.

[0157] The electronic device 100 can also store the correspondence between gestures and operations.

[0158] For example, Table 2 shows a correspondence between gestures and operations provided in an embodiment of this application.

[0159] Table 2

[0160] As shown in Table 2, the electronic device 100 can store the action components of one or more gestures, and the correspondence between each gesture and operation. For example, gesture G1 may include multiple actions executed in the following order: action P1-action P2; gesture G2 may include multiple actions executed in the following order: action P1-action P3-action P2; gesture G3 may include multiple actions executed in the following order: action P2-action P1; gesture G4 may include an action executed in the following order: action P4. Wherein, the operation corresponding to gesture G1 is operation D1; the operation corresponding to gesture G2 is operation D2; the operation corresponding to gesture G3 is operation D3; and the operation corresponding to gesture G4 is operation D4.

[0161] It is understood that the embodiments shown in Table 2 are only examples. In the embodiments of this application, the electronic device 100 may store more, fewer, or different gestures than those shown in Table 2, as well as the corresponding operations of the gestures. This application does not limit the scope of the embodiments.

[0162] In this embodiment of the application, the electronic device 100 may include a motion recognition module. The motion recognition module can collect the activity signals of the user's hand (arm and fingers, etc.), extract features from the collected activity signals, and then determine whether the user has performed an action in the action library based on the extracted features and the features of different actions stored in the action library.

[0163] In one possible implementation, the action recognition module may include one or more of the following devices: an accelerometer, a gyroscope, a touch sensor, a pressure sensor, a PPG sensor, etc. The electronic device 100 can acquire the user's activity signals (e.g., accelerometer signals, gyroscope signals, pressure signals, touchscreen signals, PPG signals, etc.) through one or more of these devices, and extract features from the acquired activity signals. In this case, the features extracted from the activity signals can be features of any of the following signals: accelerometer signals, gyroscope signals, pressure signals, touchscreen signals, PPG signals, etc. Then, the electronic device 100 can determine whether the user has performed an action from the action library based on the extracted features and the features of different actions stored in the action library.

[0164] In another possible implementation, the action recognition module may include a camera. The camera can capture multiple images of the user's hands over a period of time and extract features from these images. In this case, the features extracted from the activity signal can be features extracted from the multiple hand images. Then, the electronic device 100 can determine whether the user has performed an action from the action library based on the extracted features and the features of different actions stored in the action library.

[0165] It is understood that the embodiments described here are only two examples. In the embodiments of this application, the electronic device 100 may also use more, fewer, or different devices than those in the above embodiments to collect user activity signals. This application does not limit this.

[0166] In one possible implementation, the electronic device 100 can activate the motion recognition module after powering on and keep the motion recognition module in a constantly active state. This facilitates timely recognition of the user's actions.

[0167] In another possible implementation, the electronic device 100 can activate the motion recognition module when the screen is on and deactivate it when the screen is off. This saves power consumption. The on / off state of the electronic device 100 screen can be user-triggered active screen activation or application-triggered passive screen activation. For example, application-triggered passive screen activation could be triggered by receiving an output prompt message or a new message. In one possible implementation, the on / off state of the electronic device 100 screen can be used as a trigger condition for motion recognition or gesture recognition.

[0168] In another possible implementation, the electronic device 100 can activate all components in the motion recognition module when the screen is on, in order to recognize user actions in the screen-on state. Furthermore, when the screen is off, the electronic device 100 can deactivate some components in the motion recognition module. In this case, the motion recognition module can only recognize certain gestures, including those used to trigger the screen to turn on. This not only saves power consumption but also makes it easier for users to trigger the screen to turn on via gestures.

[0169] It is understood that the embodiments described here are only three examples. In the embodiments of this application, the electronic device 100 may also turn on / off all or part of the devices in the action recognition module under different circumstances than the above embodiments. This application does not limit this.

[0170] S302. Electronic device 100 determines that the initial action of the gesture 1 performed by the user is action 1.

[0171] After detecting that the user's action is action 1 in the action library, the electronic device 100 can determine that the user is performing a gesture, and that the starting action of the gesture (also called gesture 1) is action 1.

[0172] It should be noted that when performing step S302, the electronic device 100 can determine that the gesture 1 currently performed by the user includes action 1, and action 1 is the starting action of gesture 1.

[0173] S303. Electronic device 100 outputs gesture operation prompt 1 based on action 1, gesture operation prompt 1 is used to indicate subsequent executable actions.

[0174] It should be noted that steps S303 and S304 can be executed simultaneously, or steps S303 can be executed first and then steps S304. This application does not limit the execution order of steps S303 and S304.

[0175] In some embodiments, after recognizing that a user has performed action 1, if the electronic device 100 stores one or more gestures whose starting action is action 1, the electronic device 100 can output a gesture operation prompt 1 based on the one or more gestures whose starting action is action 1 stored in the electronic device 100. The gesture operation prompt 1 may include one or more actions following action 1 in one or more gestures whose starting action is action 1. Optionally, the gesture operation prompt 1 may also include the operation corresponding to the one or more gestures whose starting action is action 1. Further optionally, the gesture operation prompt 1 may also include the executed action 1. When the gesture operation prompt 1 includes action 1, the executed action (i.e., action 1) and the unexecuted action can be displayed in different ways, such as using different colors, fonts, sizes, thicknesses, or icons. For example, if the gesture stored in the electronic device 100 with action 1 as the starting action includes gesture G5, and gesture G5 includes multiple actions executed in the following order: action 1-action 2-action 3, then after recognizing that the user has performed action 1, gesture operation prompt 1 may include action 2, and optionally, action 3. Further optionally, gesture operation prompt 1 may also include the name of the operation corresponding to gesture G5.

[0176] It is understood that the embodiments described herein are merely examples. In the embodiments of this application, the gesture operation prompts 1 output by the electronic device 100 may also include more, less, or different content than those in the above embodiments. This application does not limit the scope of the embodiments described herein.

[0177] In this embodiment, the electronic device 100 may output gesture operation prompt 1 through one or more methods such as display screen, voice broadcast, vibration, and indicator light flashing. This application does not limit the scope of the prompt. It is understood that the electronic device 100 may also use one or more methods such as display screen, voice broadcast, vibration, and indicator light flashing to indicate that action 1 has been recognized. Other action recognition methods are similar and will not be described in detail below.

[0178] S304. Electronic device 100 determines whether the user's gesture has ended.

[0179] The specific method by which the electronic device 100 determines whether the gesture has ended can be referred to the relevant description in the embodiment shown in Figure 4 below, which will not be detailed here.

[0180] If the electronic device 100 determines that the user's gesture has not ended, the electronic device 100 may perform the following step S305.

[0181] If the electronic device 100 determines that the user's gesture has ended, the electronic device 100 may perform the following step S308.

[0182] S305. Electronic device 100 continues to detect the user's next action.

[0183] The specific method by which electronic device 100 detects the user's next action can be found in the description of the action 1 detection method in step S301 above, and will not be repeated here.

[0184] S306. Electronic device 100 updates gesture 1 based on newly detected action.

[0185] After detecting that the user has performed the next action, the electronic device 100 can determine that gesture 1 also includes a newly detected action, and that the newly detected action is the last action of gesture 1 at present.

[0186] For example, if gesture 1 only includes a starting action (i.e. action 1) before executing step S305, and the action detected in step S305 is action 1, then when executing step S306, the updated gesture 1 can include the following multiple actions in the execution order: action 1 - action 2.

[0187] For example, if gesture 1 includes the following multiple actions in the execution order before step S305: action 1-action 2, and the action detected in step S305 is action 3, then when step S306 is executed, the updated gesture 1 may include the following multiple actions in the execution order: action 1-action 2-action 3.

[0188] It is understood that the embodiments described here are just two examples. In the embodiments of this application, the newly detected action by the electronic device 100 may also be an action different from the above embodiments. The gesture 1 performed by the user may also include more, fewer or different actions than the above embodiments. This application does not limit this.

[0189] S307. Electronic device 100 outputs gesture operation prompt 2 based on gesture 1, gesture operation prompt 2 is used to indicate subsequent executable actions.

[0190] In one possible implementation, after executing step S306, the electronic device 100 may first execute step S307 and then execute step S304. In another possible implementation, after executing step S306, the electronic device 100 may also execute steps S307 and S304 simultaneously.

[0191] In some embodiments, after performing step S306, the electronic device 100 can output a gesture operation prompt 2 based on the updated gesture 1 and one or more gestures stored in the electronic device 100. The gesture operation prompt 2 may include one or more actions or action strings, and optionally, may also include operations corresponding to different actions / action strings. Specifically, if gesture 1 is completely identical to the action string consisting of the preceding one or more actions in any gesture stored in the electronic device 100, then the one or more actions in the gesture operation prompt 2 may include actions following gesture 1 in that gesture; the one or more action strings in the gesture operation prompt 2 may include action strings consisting of all actions following gesture 1 in that gesture; the operations corresponding to different actions / action strings in the gesture operation prompt 2 may include the operation corresponding to that gesture. Optionally, if the gesture operation prompt 2 includes multiple actions / action strings, and each action / action string may correspond to one gesture, then the actions / action strings corresponding to each gesture may be sorted based on the execution frequency of each gesture, displaying the actions / action strings corresponding to gestures with high execution frequencies first, and the actions / action strings corresponding to gestures with low execution frequencies last.

[0192] For example, if gesture 1 includes multiple actions performed in the following order: action 1-action 2, and the electronic device 100 stores gestures whose previous actions are exactly the same as gesture 1, including gesture G5, gesture G5 includes multiple actions performed in the following order: action 1-action 2-action 3, then after recognizing that the user has performed gesture 1, gesture operation prompt 2 may include action 3. Optionally, gesture operation prompt 2 may also include the name of the operation corresponding to gesture G5.

[0193] It is understood that the embodiments described herein are merely examples. In the embodiments of this application, the gesture operation prompts 2 output by the electronic device 100 may also include more, less, or different content than those in the above embodiments. This application does not limit this.

[0194] The electronic device 100 may output gesture operation prompts 2 through one or more methods such as display screen, voice broadcast, vibration, and indicator light flashing, but this application does not limit it.

[0195] It should be noted that during subsequent user gesture execution, the electronic device 100 can also output new gesture operation prompts in real time based on one or more actions already performed by the user. The specific content and output method of the gesture operation prompts can be referred to the relevant descriptions of gesture operation prompt 1 or gesture operation prompt 2, which are not limited here.

[0196] S308. Electronic device 100 determines gesture 1 and determines whether there is an operation corresponding to gesture 1.

[0197] Electronic device 100 can determine whether an operation corresponding to gesture 1 is stored based on the stored correspondence between gestures and operations.

[0198] If there is an operation corresponding to gesture 1, the electronic device 100 can perform the following step S309.

[0199] If there is no operation corresponding to gesture 1, in some embodiments, the electronic device 100 may perform the following step S310.

[0200] In this application embodiment, the electronic device 100 may use one or more methods such as display screen, voice broadcast, vibration, indicator light flashing to provide feedback on whether there is an operation corresponding to gesture 1. This application does not limit this.

[0201] S309. Electronic device 100 responds to gesture 1 and executes operation 1 corresponding to gesture 1.

[0202] In this embodiment, operation 1 can be an operation performed by the electronic device 100 itself, or an operation that controls other electronic devices (such as electronic device 200) to perform a specified operation. That is, the electronic device 100 sends a specific instruction to other electronic devices (such as electronic device 200), and this specific instruction can be used to instruct the receiving electronic device to perform the specified operation. In other embodiments, the electronic device 100 can also control the electronic device 200 to perform a specified operation through gestures based on the device type of the electronic device 200. The specific process can be referred to the relevant description in the embodiment shown in Figure 5 below, which will not be repeated here.

[0203] For example, if the operation 1 corresponding to gesture 1 is to open the payment code, then in response to gesture 1, electronic device 100 can open the payment code.

[0204] For example, if the operation 1 corresponding to gesture 1 is to hang up the phone, then in response to gesture 1, electronic device 100 can hang up the phone.

[0205] It is understood that the embodiments described here are just two examples. In the embodiments of this application, operation 1 may be an operation different from the above embodiments, and this application does not limit it here.

[0206] S310. Electronic device 100 outputs an error message.

[0207] Step S310 is an optional operation.

[0208] In some embodiments, if the electronic device 100 detects that the gesture 1 performed by the user does not have a corresponding operation, it may either not perform any operation or return to the interface displayed by the electronic device 100 before step S301.

[0209] In other embodiments, the electronic device 100 may output an error message when it detects that the gesture 1 performed by the user does not have a corresponding operation. The error message can be used to inform the user that the currently performed gesture 1 does not have a corresponding operation. In some embodiments, the error message may also include the action of one or more gestures with a similarity greater than a similarity threshold to gesture 1 and the corresponding operation, so that the user can understand the correct gesture after performing an incorrect gesture. It is understood that the embodiments here are only examples, and in the embodiments of this application, the error message may also include more, less, or different content than the above embodiments, and this application does not limit it.

[0210] It should be noted that the similarity between two gestures can be determined based on one or more factors, such as the number of actions, the composition of the actions, and the similarity between the actions. For example, the smaller the difference in the number of actions between two gestures, the higher the similarity between them; the closer the composition of the actions, the higher the similarity; and the higher the similarity between the actions in the corresponding execution order, the higher the similarity between the two gestures. The similarity of the actions can be preset. It is understood that the embodiments described here are merely examples. In this application, the electronic device 100 can also calculate the similarity between two gestures in a different manner than in the above embodiments, and this application does not limit this.

[0211] It is understood that the embodiment shown in FIG3 is only an example. In the embodiments of this application, the electronic device 100 may also perform the gesture control method through more, fewer, or different steps than the embodiment shown in FIG3. This application does not limit it.

[0212] The gesture control method provided in this application uses actions as the basic unit of gesture recognition, achieving gesture recognition based on action recognition, thereby creating many different gestures from a limited set of actions. This facilitates triggering more diverse functions through gestures, reducing memory pressure. Furthermore, it can also recognize continuous gestures.

[0213] The following describes the specific process by which an electronic device 100, according to an embodiment of this application, determines whether a user's gesture has been completed.

[0214] Figure 4 shows a flowchart of a method for determining whether a user's gesture has been completed, according to an embodiment of this application.

[0215] As shown in Figure 4, a specific process for determining whether a user's gesture has been completed may include the following steps:

[0216] S401. Electronic device 100 determines whether there is a gesture in the pre-stored gestures that is completely consistent with gesture 1 and has more actions than gesture 1.

[0217] Step S401 is an optional step.

[0218] If there exists a sequence of actions consisting of one or more preceding actions that is completely identical to gesture 1, and the number of actions is greater than the number of actions in gesture 1, then the user may execute that gesture. In this case, the electronic device 100 may execute the following step S402.

[0219] If there is no action sequence consisting of one or more preceding actions that is completely identical to gesture 1 and has more actions than gesture 1, then even if the user continues to perform other actions, there will be no corresponding gesture. In this case, the electronic device 100 can execute the following step S404 to determine that the user's gesture has ended.

[0220] In another possible implementation, the electronic device 100 may also perform the following step S405 to determine that the user's gesture has not ended when it detects that the current gesture 1 is not any of the pre-stored gestures, and that among the pre-stored gestures, there is an action string consisting of at least one or more actions of the preceding gesture that is completely consistent with gesture 1, and the number of actions of the gesture is greater than the number of actions of gesture 1.

[0221] S402. Electronic device 100 sets a timer with a duration of 1.

[0222] Duration 1 can be a preset duration, such as 500 milliseconds (ms), 300ms, etc.

[0223] S403. Electronic device 100 determines whether the user has been detected to perform the next action before the timer expires.

[0224] Electronic device 100 can determine whether it can detect the user performing the next action before the timer's countdown ends.

[0225] In one possible implementation, when electronic device 100 detects that the user is performing the next action, it means that electronic device 100 detects that the action performed by the user during the time period (i.e., the time period before the countdown ends) is an action from the action library stored by electronic device 100.

[0226] If the electronic device 100 detects that the user is performing the next action before the timer expires, the electronic device 100 performs the following step S405.

[0227] If the electronic device 100 does not detect the user performing the next action before the timer expires, the electronic device 100 executes the following step S404.

[0228] S404. Electronic device 100 determines that the user's gesture has ended.

[0229] S405. Electronic device 100 determines that the user's gesture has not ended.

[0230] In this way, the electronic device 100 can determine whether the user's gesture has been completed, and thus determine whether to execute the operation corresponding to the current gesture.

[0231] In other embodiments, the electronic device 100 may also determine whether the user's gesture has been completed based on the number of actions in the action sequence already executed by the user. For example, if the number of actions already executed by the user is greater than or equal to the number of actions in the gesture with the most actions in the action library, then it is determined that the user's gesture has been completed; if the number of actions already executed by the user is less than the number of actions in the gesture with the most actions in the action library, then it is determined that the user's gesture has not been completed. Optionally, based on determining that the user's gesture has not been completed based on the number of actions in the action sequence already executed by the user, the electronic device 100 may also determine whether the user's next action can be detected within a fixed time interval (e.g., duration 1) after the end of the previous action, and determine whether the user's gesture has ended based on the detection result. The specific determination method and determination result can be referred to the relevant description in the embodiment shown in Figure 4 above, and will not be repeated here.

[0232] It is understood that the embodiments described here and the embodiments shown in Figure 4 above are just two examples. In the embodiments of this application, the electronic device 100 may also use a different method than the above embodiments to determine whether the user's gesture has ended. This application does not limit this.

[0233] In some application scenarios, electronic device 100 can establish a communication connection with other electronic devices (such as electronic device 200). In this case, electronic device 100 stores the correspondence between gestures and operations performed by electronic device 200. When electronic device 100 detects that a user performs a gesture to trigger electronic device 200 to perform a specified operation, electronic device 100 can send an instruction to electronic device 200, which can be used to instruct electronic device 200 to perform the specified operation.

[0234] For example, Figure 5 shows a schematic diagram of the process by which an electronic device 100 controls an electronic device 200 to perform operations based on a recognized gesture, according to an embodiment of this application.

[0235] As shown in Figure 5, the specific process by which electronic device 100 controls electronic device 200 to perform operations based on recognized gestures may include the following steps:

[0236] S501. Electronic device 100 establishes a communication connection with electronic device 200.

[0237] Electronic device 100 can establish a communication connection with electronic device 200, which can be a wired communication connection or a wireless communication connection.

[0238] S502. Electronic device 100 obtains the device type of electronic device 200.

[0239] In some embodiments, electronic device 100 may obtain the device type of electronic device 200 before establishing a communication connection with electronic device 200, or during the process of establishing a communication connection.

[0240] In other embodiments, electronic device 100 can obtain the device type of electronic device 200 through the communication connection after establishing a communication connection with electronic device 200.

[0241] S503. Electronic device 100 determines the correspondence between gestures and operations of electronic device 200 based on the device type of electronic device 200.

[0242] In some embodiments, steps S503 and S504 may be executed simultaneously, or steps S503 may be executed first and then steps S504 may be executed, or steps S504 may be executed first and then steps S503 may be executed. This application does not limit the execution order between steps S503 and S504.

[0243] Electronic device 100 can store one or more correspondences between gestures and operations. These operations can include those performed by electronic device 100 itself, as well as those performed by other electronic devices. Electronic device 100 can store the correspondences between gestures and operations performed by other electronic devices based on their device types (or device identifiers). The correspondences between gestures and operations can be pre-stored or determined and stored based on user-defined gesture-operation correspondences. Specific methods for user-defined gesture-operation correspondences can be found in the descriptions in the embodiments shown in Figures 6A-9 below, and will not be detailed here.

[0244] In one possible implementation, the correspondence between gestures and operations stored in electronic device 100 can be a fixed relationship. In this case, different gestures correspond to different operations, and there are one or more gestures that can correspond to operations of electronic device 200. When electronic device 100 detects that the user performs any of the one or more gestures, it can trigger electronic device 100 to send a command to electronic device 200, which can be used to instruct electronic device 200 to perform the operation of electronic device 200 corresponding to the gesture.

[0245] For example, when the electronic device 100 stores a fixed correspondence between gestures and operations, the correspondence between gestures and operations is shown in Table 3.

[0246] Table 3

[0247] As shown in Table 3, the electronic device 100 can store one or more gesture action components, as well as the correspondence between each gesture and operation. Operations can include the electronic device 100 performing actions and the electronic device 100 controlling other electronic devices to perform actions. For example, gesture G6 can include multiple actions executed in the following order: action P1-action P4; gesture G7 can include multiple actions executed in the following order: action P4-action P3-action P2; gesture G8 can include an action executed in the following order: action P5. The operation of the electronic device 100 corresponding to gesture G6 can be operation D5; the operation of the electronic device of device type S1 corresponding to gesture G6 can be operation M1; the operation of the electronic device 100 corresponding to gesture G7 can be operation D6; the operation of the electronic device 100 corresponding to gesture G8 can be operation D7; and the operation of the electronic device of device type S2 corresponding to gesture G8 can be operation M2. It should be noted that, in this case, operation D5 can be sending a command to an electronic device of device type S1 to instruct that device to perform operation M1; operation D7 can be sending a command to an electronic device of device type S2 to instruct that device to perform operation M2. Furthermore, gesture G7 does not correspond to the operations of other electronic devices.

[0248] It is understood that the embodiments shown in Table 3 are only examples. In the embodiments of this application, the electronic device 100 may store more, fewer, or different gestures than those shown in Table 3, as well as the corresponding operations of the gestures. This application does not limit the scope of the embodiments.

[0249] In some embodiments, where the correspondence between gestures and operations stored in electronic device 100 is a fixed relationship, electronic device 100 can, after detecting the establishment of a communication connection with electronic device 200, filter out the correspondence between gestures and operations performed by electronic device 200 from the correspondence between gestures and operations stored in electronic device 100, based on the device type of electronic device 200.

[0250] In one possible implementation, the electronic device 100 may store the correspondence between gestures and operations in different connection states.

[0251] For example, Table 4 shows the correspondence between gestures and operations in different connection states of an electronic device 100 provided in the embodiments of this application.

[0252] Table 4

[0253] As shown in Table 4, electronic device 100 can store one or more gesture action components, and the correspondence between gestures and operations when electronic device 100 is in different connection states. For example, electronic device 100 can store gestures G6 and G7. Gesture G6 can include multiple actions executed in the following order: action P1-action P4; gesture G7 can include multiple actions executed in the following order: action P4-action P3-action P2. When electronic device 100 establishes a communication connection with electronic device of device type S1, the correspondence between gestures and operations is as follows: the operation of electronic device 100 corresponding to gesture G6 can be operation D5, and the operation of electronic device of device type S1 corresponding to gesture G6 can be operation M1; the operation of electronic device 100 corresponding to gesture G7 can be operation D6. Among them, operation D5 can be sending an instruction to electronic device 200 to instruct electronic device of device type S1 to execute operation M1; there is no correspondence between gesture G7 and operation of electronic device 200. When electronic device 100 and electronic device of device type S1 have not established a communication connection, the correspondence between gestures and operations is as follows: the operation of electronic device 100 corresponding to gesture G6 can be operation D8; the operation of electronic device 100 corresponding to gesture G7 can be operation D6. Operation D8 is different from operation D5, and operation D8 is unrelated to electronic device of device type S1.

[0254] It is understood that the embodiments shown in Table 4 are merely illustrative. When the electronic device 100 is in different connection states, the same gesture may correspond to different operations. In the embodiments of this application, the electronic device 100 may store more, fewer, or different gestures than the embodiments shown in Table 4, as well as the corresponding operations of the gestures. It may also store more, fewer, or different connection states than the above embodiments (e.g., whether to establish a communication connection with the electronic device of device type S2, etc.). This application does not limit this.

[0255] In some embodiments, where the electronic device 100 stores a correspondence between gestures and operations related to the connection state of the electronic device 100, the electronic device 100 can, after detecting the establishment of a communication connection with the electronic device 200, determine the correspondence between gestures and operations of the electronic device 200 based on the device type of the electronic device 200 from the correspondence between gestures and operations stored in the electronic device 100.

[0256] S504. Electronic device 100 detects user gesture 2.

[0257] Gesture 2 can be a gesture stored in electronic device 100.

[0258] S505. Electronic device 100 determines that the operation corresponding to gesture 2 is operation 2 of electronic device 200 based on the correspondence between gesture and operation of electronic device 200.

[0259] After recognizing the user's gesture 2, the electronic device 100 can determine whether gesture 2 can be used to trigger the electronic device 100 to control the electronic device 200 to perform an operation based on the correspondence between the gesture and the operation of the electronic device 200.

[0260] If an electronic device 100 and an electronic device 200 have established a communication connection, and the operation corresponding to gesture 2 includes operation 2 performed by electronic device 200, then electronic device 100 can perform the following step S506.

[0261] If electronic device 100 and electronic device 200 have established a communication connection, and the operation corresponding to gesture 2 does not include the operation performed by electronic device 200, then electronic device 100 can perform the operation corresponding to gesture 2.

[0262] S506. Electronic device 100 sends instruction 1 to electronic device 200, instruction 1 instructing electronic device 200 to perform operation 2.

[0263] S507. Electronic device 200 responds to instruction 1 and executes operation 2.

[0264] Electronic device 200 can receive and respond to instruction 1 sent by electronic device 100 to perform operation 2.

[0265] It is understood that the embodiment shown in Figure 5 is only an example. In the embodiments of this application, the electronic device 100 can also control more or fewer electronic devices than the embodiment shown in Figure 5 or different from the above embodiments through gestures. This application does not limit the number or type of electronic devices controlled by the electronic device 100 through gestures.

[0266] Using the gesture control method provided in the embodiments of this application, electronic device 100 can control one or more other electronic devices by recognizing user gestures.

[0267] In some embodiments, when electronic device 100 establishes communication connections with electronic devices of different device types, the same gesture can control the different electronic devices to perform different operations. For example, when electronic device 100 establishes a communication connection with a smart speaker, a gesture (two upward swipes) can control the smart speaker to increase its volume, and a gesture (two downward swipes) can control the smart speaker to decrease its volume; when electronic device 100 establishes a communication connection with a smart light, a gesture (two upward swipes) can also control the smart light to increase its brightness, and a gesture (two downward swipes) can control the smart light to decrease its brightness, and so on. It is understood that the embodiments described here are merely illustrative of how the same gesture can control different electronic devices to perform different operations when electronic device 100 is connected to electronic devices of different device types. In this application, electronic device 100 may connect to more, fewer, or different electronic devices than in the above embodiments, and the gestures performed by the user may also be different from those in the above embodiments, and the corresponding operations may also be different from those in the above embodiments. This application does not impose any limitations on these embodiments.

[0268] It should be noted that when electronic device 100 is connected to multiple other electronic devices (such as electronic device 200 and electronic device 300), and the same gesture (such as gesture G9) can control electronic device 200 to perform operation A and electronic device 300 to perform operation B.

[0269] In this scenario, in one possible implementation, when electronic device 100 detects that the user has performed gesture G9, electronic device 100 can determine the higher-priority electronic device from electronic devices 200 and 300 based on preset priorities among different device types, and determine that the control object of this gesture G9 is the higher-priority electronic device. It then sends an instruction to that electronic device to instruct it to perform the operation corresponding to gesture G9. Here, electronic device 100 can pre-store priorities for different device types; these priorities can be factory-set by electronic device 100 or user-defined.

[0270] In another possible implementation of the above scenario, when electronic device 100 detects that the user has performed gesture G9, electronic device 100 can also detect the current scene and, based on the current scene, the device type of electronic device 200, and the device type of electronic device 300, determine the control object of this gesture G9, and send an instruction to the electronic device to instruct it to perform the operation corresponding to gesture G9. For example, if electronic device 100 is connected to a smart TV and a robot vacuum cleaner, gesture G9 can control the smart TV to adjust the video playback speed, and can also control the robot vacuum cleaner to turn on or off. If electronic device 100 detects that the current scene is a user watching television, then in that scene, when electronic device 100 detects that the user has performed gesture G9, it can determine that the control object of this gesture G9 is the smart TV. In this case, electronic device 100 can send an instruction to the smart TV to instruct it to adjust the video playback speed.

[0271] It is understood that the embodiments described here are only three examples. In the embodiments of this application, if the electronic device 100 is connected to multiple other electronic devices, the electronic device 100 may also use a different method than the above embodiments to determine the object of the user's gesture. This application does not limit this.

[0272] In some application scenarios, users can create and store gestures based on the action library stored in the electronic device 100. In this way, users can combine preset actions to obtain new gestures.

[0273] For example, Figure 6A shows a schematic diagram of a gesture creation process provided in an embodiment of this application.

[0274] As shown in Figure 6A, the specific process of creating a gesture by electronic device 100 may include the following steps:

[0275] S601. Electronic device 100 receives and responds to the user's operation of creating a new gesture 3, and displays a gesture creation interface, which includes multiple action options, each action option corresponding to an action in the action library.

[0276] For example, the user's operation of creating a new gesture 3 can be the user's click operation on the added gesture control 1301 in the embodiment shown in Figure 13A below.

[0277] For example, the specific content of the gesture creation interface can be referred to the relevant descriptions in the embodiments shown in Figures 13B-13E below, or Figures 15C-15E.

[0278] In some embodiments, the action options displayed in the gesture creation interface may be determined based on the historical recognition records of the electronic device 100. For example, the electronic device 100 may determine one or more actions that have achieved a preset number of successful recognitions based on historical recognition records, and display the action options corresponding to those one or more actions in the gesture creation interface. As another example, the electronic device 100 may determine one or more actions that have achieved a preset success rate based on historical recognition records, and display the action options corresponding to those one or more actions in the gesture creation interface, and so on.

[0279] It is understood that the embodiments described here are only two examples. In the embodiments of this application, the electronic device 100 may also use other methods to determine the action options displayed in the gesture creation interface, and this application does not limit it here.

[0280] S602. Electronic device 100 receives and responds to the user's operation on the gesture creation interface, and determines gesture 3.

[0281] In some embodiments, user actions on the gesture creation interface may include one or more actions. It should be noted that during user actions on the gesture creation interface, the interface can update the displayed content (e.g., action options) in real time based on the user's actions.

[0282] For example, the action options in the gesture creation interface may include action option 1, action option 2, and action option 3, where action option 1 corresponds to action 1, action option 2 corresponds to action 2, and action option 3 corresponds to action 3. In this case, Figure 6B shows a flowchart illustrating how an electronic device 100, according to an embodiment of this application, determines gesture 3 based on multiple user operations on the gesture creation interface.

[0283] As shown in Figure 6B, the specific process by which electronic device 100 determines gesture 3 based on multiple user operations on the gesture creation interface may include the following:

[0284] S602a. Electronic device 100 receives and responds to the user's selection of action option 1, and determines that the initial action of gesture 3 is action 1.

[0285] For example, taking the embodiments shown in Figures 13B-13E as an example, action option 1 can be the fist-clenching option 1311 in the embodiment shown in Figure 13B, and action 1 can be clenching a fist.

[0286] S602b. Electronic device 100 receives and responds to the user's selection of action option 2, and determines the second action of gesture 3 as action 2.

[0287] For example, taking the embodiments shown in Figures 13B-13E as an example, action option 2 can be the joint down stroke option 1322 in the embodiment shown in Figure 13C, and action 2 can be a joint down stroke.

[0288] S602c. Electronic device 100 receives and responds to the user's selection of action option 3, and determines the third action of gesture 3 as action 3.

[0289] For example, taking the embodiments shown in Figures 13B-13E as an example, action option 2 can be release option 1332 in the embodiment shown in Figure 13D, and action 3 can be release.

[0290] S602d. Electronic device 100 receives and responds to the operation created by the user, and determines that the action string corresponding to gesture 3 is action 1-action 2-action 3.

[0291] For example, taking the embodiments shown in Figures 13B-13E as an example, the operation completed by the user can be the user's click operation on the control 1343 in the embodiment shown in Figure 13E, and the action sequence corresponding to gesture 3 can be clenching a fist - sliding down the knuckle - releasing.

[0292] It is understood that the embodiment shown in Figure 6B is merely an illustrative example. The electronic device 100 can determine the gesture 3 based on multiple operations performed by the user on the gesture creation interface. In this embodiment, the user's operations on the gesture creation interface may include more, fewer, or different operations than those in the above embodiments. This application does not limit these operations.

[0293] S603. Electronic device 100 displays a gesture test interface, which is used to indicate the progress of the gesture test.

[0294] Gesture testing progress can be used to indicate the recognition progress of a gesture test, that is, the recognition status of each action in the gesture. The recognition status can include pending recognition, successful recognition, and recognition failure.

[0295] For example, the gesture testing interface can be referred to the relevant description in the embodiments shown in Figures 13F-13I below.

[0296] S604. Electronic device 100 detects the actions performed by the user and updates the gesture test progress.

[0297] While displaying the gesture test interface, the electronic device 100 can detect the actions performed by the user and update the recognition status of each action in the gesture based on the action recognition results, that is, update the gesture test progress.

[0298] For example, the interface for updating the gesture test progress based on the detected action recognition results of electronic device 100 can also refer to the changes in the recognition identifier of each action in the embodiments shown in Figures 13F-13I below.

[0299] S605. Electronic device 100 determines whether the gesture test was successful.

[0300] A successful gesture test means that the electronic device 100 successfully recognizes each movement in the gesture sequentially.

[0301] Gesture test failure means that the electronic device 100 fails to recognize any action during the process of sequentially recognizing each action in the gesture.

[0302] If the gesture test is successful, the electronic device 100 can perform the following step S608.

[0303] If the gesture test fails, in some embodiments, the electronic device 100 may perform step S606; in other embodiments, the electronic device 100 may perform step S607.

[0304] S606. Electronic device 100 determines whether the number of test failures exceeds a preset threshold.

[0305] Step S606 is an optional step.

[0306] For example, the preset number of times threshold can be 3 or 5 times.

[0307] In one possible implementation, if the electronic device 100 determines that the number of test failures is less than or equal to a preset threshold, the electronic device 100 can execute the above step S603 to restart the test of the gesture, that is, to sequentially identify each action in the gesture starting from the initial action of the gesture.

[0308] In another possible implementation, if the electronic device 100 determines that the number of test failures is less than or equal to a preset threshold, the electronic device 100 can execute the above step S604, that is, starting from the action that failed, sequentially identify each subsequent action.

[0309] If the number of test failures exceeds a preset threshold, the electronic device 100 may execute the following step S607.

[0310] S607. Electronic device 100 outputs a creation failure message, which is used to indicate to the user that gesture 3 failed to create.

[0311] The electronic device 100 can output a creation failure message using one or more methods such as display screen, voice broadcast, vibration, and indicator light flashing.

[0312] For example, the creation failure message displayed by the electronic device 100 via a screen can be referred to the relevant description in the embodiment shown in Figure 13L below.

[0313] S608. Electronic device 100 outputs a creation success message, which is used to notify the user that gesture 3 has been created successfully.

[0314] The electronic device 100 can output a success message through one or more methods, such as display screen, voice broadcast, vibration, or indicator light flashing.

[0315] For example, the creation success message displayed by the electronic device 100 on the screen can be referred to the relevant description in the embodiment shown in Figure 13J below.

[0316] Using the gesture control method provided in this application embodiment, the electronic device 100 can create gestures based on the action library through the user's gesture creation operation. In this way, the limited actions in the action library can form multiple different gestures to match different operations, which can reduce the memory pressure of the electronic device 100.

[0317] In some application scenarios, electronic device 100 can select a gesture from user-created gestures and / or gestures set by the electronic device 100 at the factory, and set the corresponding operation for that gesture. In this way, electronic device 100 can first create (or select) a gesture, and then set the corresponding operation for that gesture, so that users can set the correspondence between gestures and operations according to their own preferences.

[0318] For example, Figure 7 shows a flowchart of setting an operation corresponding to a gesture according to an embodiment of this application.

[0319] As shown in Figure 7, the specific process of setting the operation corresponding to gesture 3 on electronic device 100 may include the following steps:

[0320] S701. Electronic device 100 detects that gesture 3 has been created / receives user selection of the created gesture 3.

[0321] In some embodiments, when a gesture (e.g., gesture 3) is created, the electronic device 100 may perform the following step S702 to prompt the user to set the operation corresponding to the gesture 3.

[0322] In other embodiments, the electronic device 100 may display one or more created gestures (or gestures set by the factory default of the electronic device 100). When the electronic device 100 receives a user's selection of one of the gestures (e.g., gesture 3), it may perform the following step S702 to prompt the user to set the operation corresponding to gesture 3.

[0323] S702. Electronic device 100 displays an operation selection interface, which is used to set the operation corresponding to gesture 3.

[0324] In some embodiments, the operation selection interface may include one or more operation options, each operation option may correspond to an operation. Each operation option may be used to trigger the electronic device 100 to set the operation corresponding to the operation option to the operation corresponding to the gesture 3.

[0325] In other embodiments, the operation option interface may also include an operation input control, which can be used to trigger the electronic device 100 to set the operation input by the user as the operation corresponding to the gesture 3.

[0326] In other embodiments, the operation selection interface may include one or more operation options, as well as operation input controls.

[0327] It is understood that the embodiments described here are only three examples. In the embodiments of this application, the operation selection interface may also include more, fewer, or different controls than those in the above embodiments, and this application does not limit it here.

[0328] For example, the operation selection interface can be referred to the relevant description in the embodiment shown in Figure 14B below.

[0329] S703. Electronic device 100 receives and responds to the user's operation on the operation selection interface, and determines that the operation corresponding to gesture 3 is operation 3.

[0330] In some embodiments, the operation selection interface may include one or more operation options. In this case, the user's operation on the operation selection interface may be an operation on any one of the operation options (e.g., the operation option corresponding to operation 3). At this time, the electronic device 100 can set the operation 3 corresponding to the operation option selected by the user as the operation corresponding to gesture 3.

[0331] In other embodiments, the operation option interface may also include an operation input control. In this case, the user's operation on the operation selection interface can be an operation on the operation input control, such as entering the name of operation 3. In this case, the electronic device 100 can set the user-input operation 3 as the operation corresponding to gesture 3.

[0332] In other embodiments, the operation selection interface may include one or more operation options and operation input controls. In this case, the electronic device 100 may set the operation corresponding to gesture 3 based on the user's selection of the operation option; or it may set the operation corresponding to gesture 3 based on the user's input operation to the operation input controls.

[0333] It is understood that the embodiments described here are only three examples. In the embodiments of this application, the operation selection interface may also include more, fewer, or different controls than those in the above embodiments. The electronic device 100 may also set the operation corresponding to gesture 3 in a different way than those in the above embodiments. This application does not limit this.

[0334] By using the gesture control method provided in this application, users can set the correspondence between gestures and operations according to their own preferences, which can provide users with a better user experience.

[0335] In some application scenarios, electronic device 100 can first select an operation and then set the corresponding gesture for that operation. In this way, users can set the corresponding gesture for the selected operation based on the usage scenario of that operation and their own preferences, providing users with a better user experience.

[0336] For example, Figure 8 shows a flowchart of setting a gesture corresponding to an operation according to an embodiment of this application.

[0337] As shown in Figure 8, the specific process of setting the gesture corresponding to operation 4 on electronic device 100 may include the following steps:

[0338] S801. Electronic device 100 receives the operation of user-selected operation 4.

[0339] For example, the user's selection of operation 4 can be the user's operation on payment code option 1703 in the embodiment shown in Figure 17A below.

[0340] S802. Electronic device 100 displays a gesture selection interface, which includes one or more gesture options and a gesture creation control, with each gesture option corresponding to a gesture.

[0341] In the gesture selection interface, each gesture option can be used to trigger the electronic device 100 to set the gesture corresponding to the gesture option to the gesture corresponding to operation 4; the gesture creation control can be used to trigger the electronic device 100 to create a gesture corresponding to operation 4.

[0342] For example, the gesture selection interface can be referred to the relevant description in the embodiment shown in Figure 17B below.

[0343] After step S802, the electronic device 100 may perform the following steps S803 or S804.

[0344] S803. Electronic device 100 receives and responds to the user's operation of selecting the gesture option corresponding to gesture 4, and determines that the gesture corresponding to operation 4 is gesture 4.

[0345] In some embodiments, the gesture selection interface may include a gesture option corresponding to gesture 4. In this case, the electronic device 100 can receive and respond to the user's selection operation for the gesture option, and set the gesture 4 corresponding to the gesture option as the gesture corresponding to operation 4.

[0346] S804. Electronic device 100 receives and responds to user operation on gesture creation control, and creates gesture 5.

[0347] In other embodiments, the electronic device 100 may also receive and respond to a user's operation on a gesture creation control, creating a gesture 5 corresponding to operation 4.

[0348] In one possible implementation, the specific way in which the electronic device 100 creates gesture 5 can be compared with the relevant description in the embodiments shown in Figures 6A-6B above, and will not be repeated here.

[0349] After step S804, the electronic device 100 may perform the following step S805.

[0350] S805. Electronic device 100 detected that gesture 5 test was successful, confirming that the gesture corresponding to operation 4 is gesture 5.

[0351] After gesture 5 is created, electronic device 100 can test gesture 5. The testing method for gesture 5 can also refer to the relevant steps in the embodiment shown in Figure 6A above, which will not be repeated here.

[0352] After detecting that gesture 5 has been successfully tested, electronic device 100 can set the newly created gesture 5 as the gesture corresponding to operation 4.

[0353] Using the gesture control method provided in this application, the electronic device 100 can set gestures corresponding to operations. Users can set gestures corresponding to selected operations based on the usage scenario of the operation and their own preferences, thus providing users with a better user experience.

[0354] In some application scenarios, operations can be preset with security levels, which can include high security and low security levels, and optionally, medium security levels. For example, operations involving payment, such as opening a payment code, can have a high security level, while locking the screen can have a low security level. When setting the correspondence between operations and gestures on the electronic device 100, the electronic device 100 needs to ensure that the security level of the gesture matches the security level of the operation (i.e., the security level of the gesture is higher than or equal to the security level of the operation). This ensures the security of the operation and provides users with higher security.

[0355] For example, Figure 9 shows a schematic diagram of the process of setting a security level to match operation and gesture in an electronic device 100 according to an embodiment of this application.

[0356] As shown in Figure 9, the specific process for setting up electronic device 100 to match security level with operation and gesture may include the following steps:

[0357] S901. Electronic device 100 receives the operation of the user setting the correspondence between operation 6 and gesture 6.

[0358] In some embodiments, the operation by which a user sets a correspondence between operation 6 and gesture 6 can be either setting gesture 6 to the gesture corresponding to operation 6, or setting operation 6 to the operation corresponding to gesture 6. That is, the electronic device 100 can first select gesture 6 and then set the operation corresponding to gesture 6 to operation 6; or, the electronic device 100 can first select operation 6 and then set the gesture corresponding to operation 6 to gesture 6. This application does not limit this.

[0359] S902. Electronic device 100 determines whether the security level of gesture 6 is lower than the security level of operation 6.

[0360] The electronic device 100 may store different security levels for operations. In some embodiments, the security level of an operation may be preset. In other embodiments, the security level of an operation may also be determined based on the use case of the operation and / or the type of application performing the operation.

[0361] For example, Table 5 shows the correspondence between the storage operation, the application type of the operation, and the security level of an electronic device 100 provided in the embodiments of this application.

[0362] Table 5

[0363] As shown in Table 5, the electronic device 100 can store the correspondence between operations, application types for performing operations, and security levels. For example, the application type for performing operation D1 can be a payment application, and the corresponding security level for payment applications can be high security; the application type for performing operation D2 can be a system-level application, and the corresponding security level for system-level applications can be low security; the application type for performing operation D3 can be a social application, and the corresponding security level for social applications can be medium security, and so on. It should be noted that system-level applications refer to applications built into the electronic device 100's system, such as flashlight applications, weather applications, alarm clock applications, etc.

[0364] It is understood that the embodiments shown in Table 5 are merely illustrative examples, and the electronic device 100 can determine the security level of an operation based on the type of application performing the operation. In the embodiments of this application, the electronic device 100 may store more, fewer, or different operations, application types, and security levels than those shown in Table 5, and this application does not impose limitations here. Furthermore, in some other embodiments, the electronic device 100 may also store the correspondence between the usage scenario of the operation, the operation, and the security level. In this case, the electronic device 100 can also determine the security level of the operation based on the usage scenario of the operation, and this application does not impose limitations here.

[0365] The following describes how to determine the security level of gestures.

[0366] Electronic device 100 can determine the security level of a gesture based on the composition and / or number of gestures.

[0367] In one possible implementation, the electronic device 100 may be set with threshold values ​​for the number of actions corresponding to gestures of different security levels.

[0368] For example, when the number of actions in any gesture is greater than or equal to the action number threshold 1 (e.g., 4), the electronic device 100 can determine that the security level of the gesture is high security level; when the number of actions in any gesture is less than the action number threshold 1, the electronic device 100 can determine that the security level of the gesture is low security level, and so on.

[0369] For example, when the number of actions in any gesture is greater than or equal to the action number threshold 1 (e.g., 4 actions), the electronic device 100 can determine that the security level of the gesture is high security level; when the number of actions in any gesture is less than the action number threshold 1 but greater than or equal to the action number threshold 2 (e.g., 2 actions), the electronic device 100 can determine that the security level of the gesture is medium security level; when the number of actions in any gesture is less than the action number threshold 2, the electronic device 100 can determine that the security level of the gesture is low security level; and so on.

[0370] It is understood that the embodiments described herein are merely illustrative of how the security level of a gesture can be determined based on the number of gesture actions. In the embodiments of this application, the security level may include more, fewer, or different levels than those described in the above embodiments, and may also include more, fewer, or different action number thresholds than those described in the above embodiments. This application does not limit these limitations.

[0371] In another possible implementation, the electronic device 100 may also include a security level for each action in its stored action library. The electronic device 100 can determine the security level of a gesture based on the security level of the actions within the gesture.

[0372] For example, Table 6 shows the security level of an action in an action library provided in an embodiment of this application.

[0373] Table 6

[0374] As shown in Table 6, the action library stored in electronic device 100 may include action P1, action P2 and action P3, wherein action P1 has a high security level; action P2 has a low security level; action P3 has a medium security level, and so on.

[0375] It is understood that the embodiments shown in Table 6 are merely illustrative examples. The electronic device 100 may store the security levels of various actions. In the embodiments of this application, the electronic device 100 may also store more, fewer, or different actions than those in the above embodiments, and the relationship between actions and security levels may also be different from those in the above embodiments. This application does not limit these actions.

[0376] In some embodiments, when determining the security level of a gesture, the electronic device 100 can determine whether a high-security-level action exists in the gesture based on the composition of the gesture's movements. If a high-security-level action exists, the gesture is determined to be at a high security level; if no high-security-level action exists but a medium-security-level action exists, the gesture is determined to be at a medium security level; if neither a high-security-level nor a medium-security-level action exists, the gesture is determined to be at a low security level. In other words, the electronic device 100 can use the highest security level of the actions in the gesture as the security level of the gesture.

[0377] In other embodiments, when determining the security level of a gesture, the electronic device 100 may determine whether the number of high-security-level actions in the gesture is greater than a preset threshold. If the number of high-security-level actions in the gesture is greater than or equal to the threshold, the gesture is determined to be at a high security level; if the number of high-security-level actions in the gesture is less than the threshold, and high-security-level actions are present in the gesture, the gesture is determined to be at a medium security level; if no high-security-level actions are present in the gesture, the gesture is determined to be at a low security level.

[0378] It is understood that the embodiments described herein are merely illustrative examples illustrating how the security level of a gesture can be determined based on the presence and / or number of high-security-level actions within the gesture. In the embodiments of this application, the security level may include more, fewer, or different levels than those described in the above embodiments, and may also include more, fewer, or different number thresholds than those described in the above embodiments; however, this application does not impose any limitations on these aspects.

[0379] In another possible implementation, the electronic device 100 may also determine the security level of the gesture based on the number of gestures and the composition of the gestures. The specific implementation method can combine the two implementation methods mentioned above, which will not be elaborated here.

[0380] If the security level of gesture 6 is lower than the security level of operation 6, the electronic device 100 determines that the security levels of gesture 6 and operation 6 do not match. At this time, the following step S905 can be executed.

[0381] If the security level of gesture 6 is higher than or equal to the security level of operation 6, the electronic device 100 determines that the security levels of gesture 6 and operation 6 are matched. At this time, the following steps S903 or S904 can be executed.

[0382] S903. Electronic device 100 outputs an operation confirmation prompt, which is used to prompt the user to confirm whether a correspondence has been set between gesture 6 and operation 6.

[0383] Step S903 is an optional step.

[0384] For example, the operation confirmation prompt can be referred to the relevant description in the embodiment shown in Figure 14C below.

[0385] S904. Electronic device 100 stores the correspondence between gesture 6 and operation 6.

[0386] S905. Electronic device 100 outputs a safety prompt, which is used to remind the user that operation 6 requires a gesture with a higher security level.

[0387] For example, safety tips may be referred to the relevant descriptions in the embodiments shown in Figures 15B, 16B, or 16G below.

[0388] After performing step S905, the electronic device 100 may perform any one of the following steps S906 to S908 according to the user's subsequent operation.

[0389] S906. Based on the user's change of operation 6 to operation 5, when the security level of gesture 6 is detected to be no lower than that of operation 5, the electronic device 100 stores the correspondence between gesture 6 and operation 5.

[0390] In some embodiments, if the security level of gesture 6 is lower than that of operation 6, the user can change the operation corresponding to gesture 6. For example, a correspondence can be established between gesture 6 and operation 5. In this case, if the electronic device 100 detects that the security level of gesture 6 is not lower than that of operation 5, the electronic device 100 can store the correspondence between gesture 6 and operation 5. Optionally, before storing the correspondence between gesture 6 and operation 5, the electronic device 100 can also output an operation confirmation prompt, prompting the user to confirm whether to establish a correspondence between gesture 6 and operation 5.

[0391] S907. Based on the user's operation of changing gesture 6 to gesture 7, when it is detected that the security level of gesture 7 is not lower than the security level of operation 6, the electronic device 100 stores the correspondence between gesture 7 and operation 6.

[0392] In some embodiments, if the security level of gesture 6 is lower than that of operation 6, the user can change the gesture corresponding to operation 6. For example, a correspondence can be established between operation 6 and gesture 7. In this case, if the electronic device 100 detects that the security level of gesture 7 is not lower than that of operation 6, the electronic device 100 can store the correspondence between gesture 7 and operation 6. Optionally, before storing the correspondence between gesture 7 and operation 6, the electronic device 100 can also output an operation confirmation prompt, prompting the user to confirm whether to establish a correspondence between gesture 7 and operation 6.

[0393] S908. Electronic device 100 updates gesture 6 based on the user's edited gesture 6.

[0394] In some embodiments, where the security level of gesture 6 is lower than that of operation 6, the user can re-edit the action composition of gesture 6 so that the security level of the modified gesture 6 is not lower than that of operation 6. The user's editing of gesture 6 may include one or more operations.

[0395] In some embodiments, during the process of a user editing gesture 6, the electronic device 100 may prompt the user to add actions based on the security level of gesture 6, such as adding a specified number of actions (e.g., 2), adding actions of a specified security level, or adding a specified number of actions of a specified security level, etc. For example, the specific editing process of gesture 6 can be referred to the relevant descriptions in the embodiments shown in Figures 16B-16D or Figures 16G-16I below.

[0396] In other embodiments, during the user's editing of gesture 6, the electronic device 100 may also change all the action components of gesture 6, that is, recreate gesture 6 from the initial action of gesture 6. Optionally, during the creation of gesture 6, the electronic device 100 may prompt the user to select an action with a specified security level based on the security level of operation 6. Exemplarily, the specific editing process of gesture 6 can be referred to the relevant description in the embodiments shown in Figures 15B-15E below.

[0397] It should be noted that after re-editing the action composition of gesture 6, electronic device 100 can test the updated gesture 6 and determine that gesture 6 can successfully pass the test.

[0398] It is understood that the embodiments described here are just two examples. In the embodiments of this application, the electronic device 100 may also re-edit the action composition of the gesture 6 in a different manner than the above embodiments to determine the updated gesture 6.

[0399] After performing step S908, the electronic device 100 may perform the following step S909.

[0400] S909. When the electronic device 100 detects that the security level of the updated gesture 6 is not lower than the security level of the operation 6, it stores the correspondence between the gesture 6 and the operation 6.

[0401] The gesture control method provided in this application embodiment can set a correspondence between gestures and operations that match the security level based on the security level of the gestures and operations, thereby providing security for the user.

[0402] The following is an exemplary description of a user performing a gesture provided in an embodiment of this application.

[0403] Figures 10A-10D illustrate a set of user-performed gestures provided in an embodiment of this application.

[0404] As shown in Figure 10A, the user wears an electronic device 100 on their left hand, and the user's left hand performs the initial gesture: making a fist.

[0405] Within a duration of 1 (e.g., 500ms) after the user makes a fist as shown in Figure 10A, the user's left hand can perform the second action of the gesture, namely, releasing the fist, as shown in Figure 10B.

[0406] Within a time interval of 1 (e.g., 500ms) after the user releases the hand as shown in Figure 10B, as shown in Figure 10C, the user's left hand can perform the third action of the gesture, namely, clenching a fist.

[0407] Within a time interval (e.g., 500ms) after the user makes a fist as shown in Figure 10C, the user's left hand can perform the last action of the gesture, i.e., release, as shown in Figure 10D.

[0408] In this way, the user can perform a gesture by following the sequence of the four actions shown in the embodiments of Figures 10A-10D: clench fist - release - clench fist - release.

[0409] It is understood that the embodiments shown in Figures 1A-10D above are only examples. In the embodiments of this application, the electronic device 100 may store more, fewer, or different gestures than those in the above embodiments, and this application does not limit them.

[0410] The gesture control method provided in this application embodiment is described below with reference to a specific interface.

[0411] In some application scenarios, when the electronic device 100 detects that a user has performed action 1, it displays a gesture operation prompt 1. Gesture operation prompt 1 may include one or more actions / action strings, which can be used to prompt the user for actions / action strings that can be executed after action 1. Optionally, gesture operation prompt 1 may also include an operation name and / or the already executed action (i.e., action 1). The operation name can be used to indicate the operation that can be triggered by continuing to execute the one or more actions / action strings. After action 1, the electronic device 100 can detect subsequent actions performed by the user and display a gesture operation prompt 2 based on these subsequent actions. Gesture operation prompt 2 may include one or more actions / action strings, which can be used to prompt the user for actions / action strings that can be executed after the already executed action. Optionally, gesture operation prompt 2 may also include an operation name and / or the already executed action. It should be noted that gesture operation prompt 2 can be continuously updated based on the addition of actions performed by the user. After detecting that the user's gesture has been completed, the electronic device 100 can execute the operation corresponding to that gesture.

[0412] In this way, the electronic device 100 can complete gesture recognition based on action recognition, execute the operation corresponding to the gesture, and also prompt the user with subsequent executable actions / action sequences and corresponding operations through gesture operation prompts (such as gesture operation prompt 1, gesture operation prompt 2, etc.) during the gesture execution process.

[0413] Figures 11A-11J show a set of schematic diagrams of an interface for answering / hanging up a phone call via gestures, provided in an embodiment of this application.

[0414] For example, as shown in FIG11A, the electronic device 100 is in a partially off state. In the partially off state, the electronic device 100 may display the current time (e.g., 08:00) in a part of the display screen, and optionally, may also display the date (e.g., Thursday and 11 / 19).

[0415] When electronic device 100 receives a call from another electronic device, it can display the incoming call interface 1100 as shown in Figure 11B.

[0416] As shown in Figure 11B, the incoming call interface 1100 may include incoming call information 1101, which may include one or more of the following: caller (e.g., mother), caller ID, etc. The incoming call interface 1100 may also include an answer control 1102 and a hang-up control 1103. The answer control 1102 can be used to trigger the electronic device 100 to answer the call, and the hang-up control 1103 can be used to trigger the electronic device 100 to hang up the call. Optionally, the incoming call interface 1100 may also include a gesture prompt 1104, which can be used to indicate the selectable gestures and corresponding operations in the context of an incoming call. The gesture prompt 1104 may be text, such as "Gesture prompt: Clench your fist - Release twice to answer the call" and "Clench your fist - Release three times to hang up the call".

[0417] When the electronic device 100 detects that the user's first action is a fist clenching action, the electronic device 100 can display the gesture operation prompt interface 1110 as shown in Figure 11C.

[0418] As shown in Figure 11C, the gesture operation prompt interface 1110 may include gesture operation prompts, which may include executed actions 1111 and optional actions 1112. Executed actions 1111 can be used to prompt the user about the currently performed action, such as clenching a fist. Optional actions 1112 can be used to prompt the user about subsequent actions or action sequences, such as releasing. It should be noted that there is continuity between the first action indicated by optional actions 1112 and the last action indicated by executed actions 1111; that is, after performing the last action of executed actions, the user's limbs are capable of performing the first action in optional actions 1112. For example, when the last action of executed actions is clenching a fist, the first action of optional actions cannot be pinching, etc. In some embodiments, executed actions 1111 may adopt a different display method than optional actions 1112 to facilitate user differentiation. For example, executed actions 1111 may adopt a different display color than optional actions 1112, etc. This application does not limit the specific display method. Optionally, the gesture operation prompt interface 1110 may also display part of the content of the incoming call interface 1100, such as incoming call information 1101, to notify the user that the current electronic device 100 has an incoming call.

[0419] Subsequently, when the electronic device 100 detects that the user's second action is to release, as shown in Figure 11D, the electronic device 100 can update the gesture operation prompt in the gesture operation prompt interface 1110, that is, replace the executed action 1111 with the executed action 1113, and replace the optional action 1112 with the optional action 1114.

[0420] As shown in Figure 11D, the executed action 1113 can include the currently executed action string, i.e., clenching and releasing the fist; the optional action 1114 can include subsequent optional actions / action strings, i.e., clenching the fist.

[0421] Subsequently, when the electronic device 100 detects that the user's third action is to clench a fist, as shown in Figure 11E, the electronic device 100 can update the gesture operation prompt in the gesture operation prompt interface 1110, that is, replace the executed action 1113 with the executed action 1115, and replace the optional action 1114 with the optional action 1116.

[0422] As shown in Figure 11E, the executed action 1115 can include the currently executed action sequence, i.e., clench fist-release-clench fist; the optional action 1116 can include subsequent optional actions / action sequences, i.e., release.

[0423] Subsequently, when the electronic device 100 detects that the user's fourth action is to release, as shown in Figure 11F, the electronic device 100 can update the gesture operation prompt in the gesture operation prompt interface 1110, that is, replace the executed action 1115 with the executed action 1117, and replace the optional action 1116 with the optional action 1118.

[0424] As shown in Figure 11F, the executed action 1117 can include the currently executed action sequence, i.e., clench fist - release - clench fist - release; the optional action 1118 can include subsequent optional actions / action sequences, i.e., clench fist - release. Optionally, the optional action 1118 may also include an operation prompt, such as hanging up, which can be used to prompt the user that executing the action / action sequence indicated by the optional action 1118 can trigger the electronic device 100 to hang up the incoming call. Optionally, the gesture operation prompt interface 1110 may also include an end prompt 1119, which can be used to prompt the user to end the gesture, and can also prompt the user that after the current gesture ends, the electronic device 100 can be triggered to answer the incoming call.

[0425] When the electronic device 100 detects the user's ending gesture, the electronic device 100 can respond to the user's fist-open-fist-open gesture, answer the call, and display the call interface 1120 as shown in Figure 11G.

[0426] As shown in Figure 11G, the call interface 1120 may include call information 1121, which may include one or more of the following: caller (e.g., mother), call number, call duration, etc. In addition, the call interface 1120 may also include a hang-up control 1122 and a speaker control 1123. The hang-up control 1122 can be used to trigger the electronic device 100 to hang up the phone, and the speaker 1123 can be used to trigger the electronic device 100 to turn on the speaker.

[0427] During a call, when the electronic device 100 detects that the user has made a fist gesture, the electronic device 100 may display a gesture operation prompt interface 1130 as shown in Figure 11H.

[0428] As shown in Figure 11H, the gesture operation prompt interface 1130 may include gesture operation prompts for a call scenario. These prompts may include an executed action 1131 and an optional action 1132. The executed action 1131 may include the action of clenching a fist. The optional action 1132 may include a subsequent action that can be selected, such as releasing the fist. The specific functions of the executed action 1131 and the optional action 1132 can be found in the relevant descriptions in the embodiment shown in Figure 11C above, and will not be repeated here. In addition, the gesture operation prompt interface 1130 may also display call information 1121 and other content to indicate to the user that they are currently in a call scenario. Optionally, the gesture operation prompt interface 1130 may also include a gesture prompt 1133, which can be used to indicate the gestures that can be executed in the current scenario and the corresponding operations. The gesture prompt 1133 may include text, such as "Gesture prompt: Clench fist - Release three times to hang up the phone".

[0429] Subsequently, the electronic device 100 can update the gesture operation prompts based on the detected user actions. The update method can be compared with the relevant descriptions in the embodiments shown in Figures 11C-11E above, and will not be repeated here.

[0430] When the user is detected to have performed the action sequence "clench fist - release - clench fist - release - clench fist", as shown in Figure 11I, the electronic device 100 can display an updated gesture operation prompt. This prompt can include an executed action 1134 and an optional action 1135. The executed action 1134 can include the executed action sequence, i.e., clench fist - release - clench fist - release - clench fist; the optional action 1135 can include the currently executable action, i.e., release. Optionally, the optional action 1135 can also include an operation that can be triggered by executing the action indicated by the optional action 1135, i.e., hanging up. Further optionally, the gesture operation prompt interface 1130 can also include a cancellation prompt 1136, which can be used to prompt the user to cancel the hang-up, i.e., maintain the current action.

[0431] After detecting that the user has released the device, the electronic device 100 can hang up the call and display a partial screen-off interface as shown in Figure 11J. The specific description of this partial screen-off interface can be found in the relevant description in the embodiment shown in Figure 11A above, and will not be repeated here.

[0432] It is understood that the embodiments shown in Figures 11A-11J are only a set of examples. In the embodiments of this application, the electronic device 100 may also display gesture operation prompts based on the actions performed by the user in other application scenarios. Moreover, the gestures performed by the user and the corresponding operations may be different from those in the above embodiments, and this application does not limit them here.

[0433] For example, when electronic device 100 receives a call from another electronic device and displays the incoming call interface 1100 as shown in FIG11B, electronic device 100 can also display the gesture operation prompt interface 1140 as shown in FIG11K when it detects that the user's first action is a fist clenching action.

[0434] As shown in Figure 11K, the gesture operation prompt interface 1140 may include gesture operation prompts, which may include an executed action 1141 and an optional action string 1142. The executed action 1141 can be used to prompt the user about the currently performed action, such as clenching a fist. The optional action string 1142 can be used to prompt the user about the subsequent action string that can be performed, such as releasing-clenching-releasing. Optionally, the optional action string 1142 may also include an operation corresponding to the action string, such as "answering a call". It should be noted that there is a continuity between the first action indicated by the optional action string 1142 and the last action indicated by the executed action 1141; that is, after performing the last action of the executed action, the user's limbs are capable of performing the first action in the optional action string 1142. In some embodiments, the executed action 1141 may adopt a different display method than the optional action string 1142 to facilitate user differentiation. For example, the executed action 1141 may adopt a different display color than the optional action string 1142, etc. This application does not limit the specific display method. Optionally, the gesture operation prompt interface 1140 may also display part of the content of the incoming call interface 1100, such as incoming call information 1101, to notify the user that the current electronic device 100 has an incoming call.

[0435] Subsequently, when the electronic device 100 detects that the user's second action is to release, as shown in Figure 11L, the electronic device 100 can update the gesture operation prompt in the gesture operation prompt interface 1140, that is, replace the executed action 1141 with the executed action 1143, and replace the optional action string 1142 with the optional action string 1144.

[0436] As shown in Figure 11L, the executed action 1143 can include the currently executed action string, i.e., clenching and releasing a fist; the optional action string 1144 can include subsequent action strings that can be selected for execution, such as clenching and releasing a fist. It should be noted that if the second action performed by the user is the same as the first action in the optional action string 1142 shown in Figure 11K, the first action in the optional action string 1142 can be deleted to obtain the optional action string 1144; simultaneously, the second action performed by the user can be added to the executed action 1141 to obtain the executed action 1143. Optionally, the optional action string 1144 can also include the operation corresponding to the action string, such as "answering a call".

[0437] Subsequently, when the electronic device 100 detects that the user's third action is to clench a fist, as shown in Figure 11M, the electronic device 100 can update the gesture operation prompt in the gesture operation prompt interface 1140, that is, replace the executed action 1143 with the executed action 1145, and replace the optional action string 1144 with the optional action string 1146.

[0438] As shown in Figure 11M, the executed action 1145 can include the currently executed action string, i.e., clench fist - release - clench fist; the optional action string 1146 can include the action string that can be executed later, i.e., release. It should be noted that if the third action performed by the user is the same as the first action in the optional action string 1144 shown in Figure 11L, the first action in the optional action string 1144 can be deleted to obtain the optional action string 1146; simultaneously, the third action performed by the user can be added to the executed action 1143 to obtain the executed action 1145. The optional action string 1146 can also include the operation corresponding to the action string, such as "answer".

[0439] Subsequently, when the electronic device 100 detects that the user's fourth action is to release, in some embodiments, the electronic device 100 may display the gesture operation prompt interface 1110 shown in FIG11F above. In other embodiments, the electronic device 100 may also display the call interface 1120 shown in FIG11N. The specific content of the call interface 1120 can be referred to the relevant description in the embodiment shown in FIG11G above, and will not be repeated here.

[0440] It is understood that the embodiments shown in Figures 11K-11N are merely examples. In the embodiments of this application, the electronic device 100 may also display more, fewer, or different optional action sequences than the above embodiments in the case of an incoming call. This application does not impose any limitations on this. Furthermore, the electronic device 100 may also display gesture operation prompts based on user-performed actions in other application scenarios. Moreover, the gestures performed by the user and the corresponding operations may differ from those in the above embodiments. This application does not impose any limitations on this.

[0441] It is understood that, based on the above example of a call scenario, the solution of this application embodiment can also be applied to other scenarios. For example, in the scenario of an alarm clock ringing, the alarm can be canceled or delayed by gestures. For instance, the alarm can be turned off by "clenching a fist - holding for 5 seconds - releasing". In the scenario of music playback, the playback and pause can be controlled by gestures. For instance, the volume can be increased by "pressing the knuckle - swiping the knuckle up" and decreased by "pressing the knuckle - swiping the knuckle down". Similarly, the display brightness can be increased by "pressing the knuckle - swiping the knuckle up" and decreased by "pressing the knuckle - swiping the knuckle down". In the scenario of receiving a message, the message can be viewed and cleared by gestures. In the scenario of a timer, the function of pausing, continuing, and ending can be selected by gestures. This application does not limit the scenarios in which gestures can be applied.

[0442] In some application scenarios, after detecting that a user has performed an action from a motion library, the electronic device 100 can display a gesture operation prompt based on the user's performed action. The gesture operation prompt can include one or more gestures stored by the electronic device 100, where the action string consisting of the preceding one or more actions of these gestures is identical to the action string already performed by the user. Optionally, the gesture operation prompt can also include the operation corresponding to the one or more gestures. It should be noted that during the user's gesture execution, the electronic device 100 can update the gesture operation prompt in real time based on the user's performed action.

[0443] In this way, users can determine the currently available gestures and their corresponding operations based on the gesture operation prompts, preventing users from forgetting the components of the gesture during the execution process and providing a better user experience.

[0444] Figures 12A-12E illustrate a set of interface diagrams for updating gesture operation prompts based on user-performed actions, provided in an embodiment of this application.

[0445] For example, as shown in FIG12A, electronic device 100 may display a dial interface 1200, which may include time (e.g., 08:00) and date (e.g., Thursday and 11 / 19).

[0446] When the electronic device 100 detects that the user has clenched their fist, it can display the gesture operation prompt interface 1210 as shown in Figure 12B.

[0447] As shown in Figure 12B, the gesture operation prompt interface 1210 may include an executed action 1211 and one or more optional gestures, such as gesture 1212, gesture 1213, and gesture 1214. The executed action 1211 can be used to indicate an action / action sequence performed by the user, such as clenching a fist. Each optional gesture can be used to indicate a gesture stored in the electronic device 100, and the action sequence consisting of one or more preceding actions of that gesture is exactly the same as the action / action sequence performed by the user. When the executed action 1211 only includes clenching a fist, the starting action of each optional gesture is clenching a fist. Optionally, each optional gesture may also include an operation corresponding to that gesture. For example, gesture 1212 may include the action sequence "clench fist - hold - release," and may also include the operation corresponding to gesture 1212, i.e., playing music. Gesture 1213 may include the action sequence "clench fist - release - clench fist - release," and may also include the operation corresponding to gesture 1213, i.e., making a phone call. Gesture 1214 may include the action sequence “clench fist - hold - rotate wrist - release”, and may also include the operation corresponding to gesture 1214, namely, opening the payment code.

[0448] When electronic device 100 detects that the user performs the action sequence of clenching a fist and holding, since the second action of gesture 1213 is not held, as shown in Figure 12C, electronic device 100 can stop displaying gesture 1213.

[0449] When electronic device 100 detects that a user performs the action sequence clench-hold-release, since this action sequence is exactly the same as gesture 1112, in some embodiments, electronic device 100 may display a waiting interface 1220 as shown in FIG12D. In other embodiments, electronic device 100 may also display a music playback interface 1230 as shown in FIG12E.

[0450] As shown in Figure 12D, the waiting interface 1220 may include an executed gesture 1221 and a waiting prompt 1222. The executed gesture 1221 can indicate the action components of the gesture, i.e., clenching a fist, holding, and releasing. The waiting prompt 1222 can indicate to the user that the operation corresponding to the gesture is being performed and that the user needs to wait. In some embodiments, the waiting prompt 1222 may include text, such as, "Opening music software, please wait..."

[0451] After the electronic device 100 performs the operation corresponding to the gesture "clench fist-hold-release" (i.e. play music), the electronic device 100 can display the music playback interface 1230 as shown in Figure 12E.

[0452] As shown in Figure 12E, the music playback interface 1230 may include a music card 1231, which may include a music name and music playback controls, such as a previous track control, a pause / play control, and a next track control. The music playback interface 1230 can be used to play music.

[0453] It is understood that the embodiments shown in Figures 12A-12E are only a set of examples. In the embodiments of this application, the electronic device 100 may also display more, fewer or different optional gestures than those in the above embodiments based on the executed action string when it detects that the user is performing an action / action string different from the above embodiments. This application does not limit this.

[0454] In other embodiments, the electronic device 100 may also output gesture operation prompts based on the scene in which it is located. The scene in which the electronic device 100 is located may include, but is not limited to, any one or more of the following: home scene, outdoor sports scene, sleep scene, work scene, study scene, shopping mall scene, entertainment scene, etc. The electronic device 100 may determine its current scene based on its location and / or motion detection. For example, if the scene in which the electronic device 100 is located is a home scene, after detecting that the user has performed action 1, the electronic device 100 can identify from a pre-stored set of gestures that starts with action 1 and is highly relevant to the home scene, and display that gesture (or one or more actions following action 1) in the gesture operation prompt. Optionally, the gesture operation prompt may also display gestures (or actions) highly relevant to the home scene at the forefront. The relevance of a gesture to the scene can be determined based on the operation corresponding to that gesture; for example, gestures corresponding to operations related to smart lights, televisions, speakers, and other home appliances can be considered gestures highly relevant to the home scene. For example, gestures related to outdoor sports (such as monitoring exercise pace and calorie consumption) can be considered highly relevant to outdoor sports scenarios; gestures related to sleep (such as activating sleep mode and monitoring sleep quality) can be considered highly relevant to sleep scenarios; gestures related to work (such as initiating or joining a video conference, activating work mode, or turning on a projector) can be considered highly relevant to work scenarios; gestures related to learning (such as opening educational apps or viewing learning materials) can be considered highly relevant to learning scenarios; gestures related to shopping (such as making payments, searching for nearby merchants, or searching for nearby promotions) can be considered highly relevant to shopping mall scenarios; and gestures related to entertainment (such as opening games or playing movies) can be considered highly relevant to entertainment scenarios. It is understood that the embodiments described here are merely examples. In the embodiments of this application, the electronic device 100 may also determine the scene based on other methods, and the correlation between gestures (or actions) and the scene may also be determined in a different way than in the embodiments described above. This application does not limit this.

[0455] In some application scenarios, the electronic device 100 can display a gesture settings interface, which may include adding gesture controls. The electronic device 100 can receive and respond to the user's operation to add a gesture control, displaying a gesture creation interface with one or more action options, each corresponding to a gesture from a gesture library. Based on the user's operation on the gesture creation interface, the electronic device 100 can display the user-selected action string, along with one or more action options. The electronic device 100 can also receive and respond to the user's completion of the creation operation, displaying a gesture test interface to test whether the newly created gesture can be successfully recognized. After successfully recognizing the newly created gesture, the electronic device 100 determines that the gesture has been successfully added and stored.

[0456] In this way, the electronic device 100 can add multiple custom gestures based on multiple actions in the action library, enriching the diversity of gestures.

[0457] Figures 13A-13M illustrate a set of interface diagrams for creating custom gestures provided in embodiments of this application.

[0458] For example, as shown in Figure 13A, the electronic device 100 may display a gesture setting interface 1300. The gesture setting interface 1300 may include one or more gesture options, such as gesture option A, gesture option B, gesture option C, etc. Each gesture option can be used to trigger the electronic device 100 to display the editing interface of the gesture corresponding to that gesture option. The editing interface can be used to edit the action components of the gesture and / or the operation corresponding to the gesture. In addition, the gesture setting interface 1300 may also include an add gesture control 1301. The add gesture control 1301 can be used to trigger the electronic device 100 to display a gesture creation interface to create new gestures.

[0459] The electronic device 100 can receive and respond to the user's click operation on the added gesture control 1301, and display the gesture creation interface 1310 as shown in FIG13B.

[0460] As shown in Figure 13B, the gesture creation interface 1310 may include one or more action options, such as a fist option 1311, a release option, a hold option, a wrist raise option, a pinch option, a knuckle swipe up option, and a knuckle swipe down option. In one possible implementation, the first action of the gesture cannot be a continuous action; a continuous action refers to an action that does not involve opening or closing the hand, and can also be understood as maintaining or continuing the previous action, such as continuously pinching or continuously clenching a fist. That is, the hold option cannot be selected, and hold is used as the first action. In some embodiments, the electronic device 100 may default the initial state of the user's gesture to release. In this case, the action corresponding to each action option displayed in the gesture creation interface 1310 is continuous with the release action, such as the fist option 1311, the hold option, the wrist raise option, the pinch option, the knuckle swipe up option, and the knuckle swipe down option. That is, action options that are not continuous with the release action, such as the release option, may not be displayed. Optionally, when providing actions for the user to choose from, the electronic device 100 may recommend actions with high accuracy based on the historical accuracy of each action recognition. Optionally, the gesture creation interface 1310 may also display the theme of the current interface, such as "Create New Gesture D," to prompt the user that the gesture creation interface 1310 can be used to create gesture D. Optionally, each action option in the gesture creation interface 1310 may display a security level indicator, which can be used to indicate the security level of the action corresponding to that action option. In some embodiments, the security level indicator may use differences in display color, font, font size, or any one or more display methods to indicate different security levels; in other embodiments, the security level indicator may also use different symbols to indicate different security levels. This application does not limit the specific display form of the security level indicator.

[0461] Electronic device 100 can receive and respond to a user's tap on the fist option 1311, displaying a gesture creation interface 1320 as shown in Figure 13C.

[0462] As shown in Figure 13C, the gesture creation interface 1320 may include a selected action 1321 and one or more action options. The selected action 1321 may include a sequence of actions selected by the user in the order of selection, such as "clenching a fist." Furthermore, each of the one or more action options may correspond to an action from a motion library. In some embodiments, the action corresponding to each action option needs to be sequential with the last action of the selected action 1321; that is, after the user performs the last action of the selected action 1321, the user's limbs can perform the action corresponding to each action option. It should be noted that in some embodiments, the one or more action options can be determined based on the last action in the selected action 1321. When the last action in the selected action 1321 changes, the one or more action options can change accordingly, and the action corresponding to each action option is sequential with the last action of the selected action 1321. For example, as shown in the embodiment of Figure 13B, when no action is selected, the one or more action options displayed in the gesture creation interface 1310 may include a fist-clenching option 1311, a release option, a hold option, a wrist-raising option, a pinching option, a knuckle-up option, a knuckle-down option, etc. After the user selects the fist-clenching option 1311, since there is no continuity between the fist-clenching and other actions such as pinching, the one or more action options in the gesture creation interface 1320 shown in Figure 13C may include a release option, a hold option, a knuckle-up option, and a knuckle-down option 1322, etc., and the action options corresponding to the fist-clenching and pinching actions will no longer be displayed. In addition, the one or more action options displayed in the gesture creation interface 1330 may be obtained by removing action options that are not continuous with the fist-clenching action from one or more action options in the gesture creation interface 1310 shown in Figure 13B, or may include other action options that are continuous with the fist-clenching action but are not displayed in the gesture creation interface 1310. This application does not limit this. The gesture creation interface 1320 may also include a confirmation control 1323, which can be used to trigger the electronic device 100 to confirm the action string indicated by the selected action 1321 as the gesture created this time, and display the test interface of the gesture.

[0463] The electronic device 100 can receive and respond to the user's tap operation on the knuckle swipe option 1322, and display the gesture creation interface 1330 as shown in FIG13D.

[0464] As shown in Figure 13D, the gesture creation interface 1330 may include a selected action 1331 and one or more action options, such as a release option 1332, a hold option, and a knuckle swipe option. It should be noted that, at this point, the last action selected by the user is a knuckle swipe down; therefore, each action corresponding to these one or more action options is sequential with the knuckle swipe down. The gesture creation interface 1330 may also include a confirmation control 1333. The selected action 1331 may include a sequence of actions already selected by the user, namely, clenching a fist and swiping down with a knuckle. It should be noted that in the selected action 1331, clenching a fist and swiping down with a knuckle can be displayed according to the user's selection order, for example, displaying the clenched fist above (or before) the knuckle swipe down. In some embodiments, when the selected action 1331 includes multiple actions, the electronic device 100 may also adjust the order of actions in the selected action 1331 based on the user's operation of adjusting the action order; this application does not limit this. Furthermore, the specific functions of the selected action 1331, one or more action options, and the confirmation control 1333 can be found in the description of the selected action 1321, one or more action options, and the confirmation control 1323 in the embodiment shown in Figure 13C above, and will not be repeated here.

[0465] Electronic device 100 can receive and respond to a user's click operation on release option 1332, and display gesture creation interface 1340 as shown in FIG13E.

[0466] As shown in Figure 13E, the gesture creation interface 1340 may include a selected action 1341 and one or more action options, such as a fist clenching option 1342 and a pinching option. It should be noted that the last action selected by the user is pinching; therefore, each action corresponding to the one or more action options is sequential with pinching. The gesture creation interface 1340 may also include a confirmation control 1343. The selected action 1341 may include a sequence of actions already selected by the user, namely, clenching the fist, swiping down the knuckle, and releasing. It should be noted that in the selected action 1341, clenching the fist, swiping down the knuckle, and releasing can be displayed in the order selected by the user. Furthermore, the specific functions of the selected action 1341, one or more action options, and the confirmation control 1343 can be found in the description of the selected action 1321, one or more action options, and the confirmation control 1323 in the embodiment shown in Figure 13C above, and will not be repeated here.

[0467] Optionally, if the gesture already exists, i.e., the user has created a repeating gesture, the electronic device 100 may respond to the user's click operation on the confirmation control 1343 by outputting a prompt to indicate to the user that the gesture already exists.

[0468] If the gesture is not present, the electronic device 100 can receive and respond to the user's click operation on the confirmation control 1343 and display the gesture test interface 1350 as shown in FIG13F.

[0469] As shown in Figure 13F, the gesture testing interface 1350 may include one or more actions, each corresponding to an identification marker. These one or more actions include all actions in a newly created gesture by the user, and the order of these actions is the same as the order of actions in a newly created gesture. These one or more actions include clenching a fist 1351, swiping down with a knuckle 1352, and releasing 1353. Clenching a fist 1351 corresponds to an identification marker 1354, swiping down with a knuckle 1352 corresponds to an identification marker 1355, and releasing 1353 corresponds to an identification marker 1356. The identification marker may include three states: pending recognition, successful recognition, and failed recognition. The identification marker can be used to indicate whether the action has been successfully recognized by the electronic device 100. In some embodiments, when the identification marker is empty, it is in a pending recognition state; when a checkmark is displayed in the identification marker, it is in a successful recognition state; and when a cross is displayed in the identification marker, it is in a failed recognition state. In the embodiment shown in Figure 13F, the identification markers for all actions are in a pending recognition state.

[0470] After detecting that the user has clenched their fist, as shown in Figure 13G, the electronic device 100 can switch the identification mark 1354 of the clenched fist 1351 from the pending identification state to the identification successful state.

[0471] After detecting that the user has performed a knuckle swipe, as shown in Figure 13H, the electronic device 100 can switch the identification mark 1355 of the knuckle swipe 1352 from the pending identification state to the identification successful state.

[0472] After detecting that the user has released the device, in some embodiments, as shown in FIG13I, the electronic device 100 can switch the identification mark 1356 of the release 1353 from the pending identification state to the identification success state. In other embodiments, the electronic device 100 may also display a creation success message as shown in FIG13J.

[0473] After detecting that all actions in the newly created gesture have been successfully recognized, the electronic device 100 can display a creation success interface 1360 as shown in Figure 13J.

[0474] As shown in Figure 13J, the creation success interface 1360 may include a creation success prompt 1361, which can be used to notify the user that the gesture has been created successfully. In some embodiments, the creation success prompt 1361 may include text, such as "New gesture D created successfully".

[0475] In other application scenarios, if the electronic device 100 fails to recognize the gesture during the gesture test, and the number of failures is less than a preset threshold, the electronic device 100 can prompt the user that the test has failed and restart the test of all actions of the gesture.

[0476] For example, if the electronic device 100 successfully recognizes a clenched fist and a downward swipe of the knuckle during the gesture test, but fails to recognize a release, the electronic device 100 can display a gesture test interface 1350 as shown in Figure 13K. In the gesture test interface 1350 shown in Figure 13K, recognition markers 1354 and 1355 are both in a successful recognition state, while recognition marker 1356 is in a failed recognition state.

[0477] When the electronic device 100 detects that any action recognition fails, the electronic device 100 can display a failure prompt interface 1370 as shown in Figure 13L.

[0478] As shown in Figure 13L, the failure message interface 1370 may include a failure message 1371, which can be used to inform the user that gesture creation failed. In some embodiments, the failure message 1371 may include text, such as "Creation failed, please try again!" Furthermore, the failure message interface 1370 may also include a retest control 1372. The retest control 1372 can be used to trigger the electronic device 100 to redisplay the gesture test interface 1350.

[0479] In some embodiments, the electronic device 100 may receive and respond to a user's click operation on the retest control 1372, and display the gesture test interface 1350 as shown in FIG13M. Optionally, the electronic device 100 may display the gesture test interface 1350 as shown in FIG13M when it is determined that the number of gesture recognition failures is less than a preset threshold number.

[0480] As shown in Figure 13M, a detailed description of the gesture testing interface 1350 can be found in the relevant description in the embodiment shown in Figure 13F above, and will not be repeated here. It should be noted that in the gesture testing interface 1350, all action recognition markers are in a pending recognition state, that is, after a gesture recognition failure, the electronic device 100 can restart the recognition from the first action of the gesture.

[0481] In other embodiments, the electronic device 100 may also receive and respond to a user's click on the retest control 1372, and redisplay the gesture test interface 1350 as shown in FIG13H. That is, the electronic device 100 may start recognizing from the failed gesture after determining that gesture recognition has failed.

[0482] In some embodiments, if the electronic device 100 determines that the number of gesture recognition failures is greater than or equal to a preset threshold, the electronic device 100 can return to the gesture creation interface 1310 shown in FIG13B and recreate the gesture.

[0483] It is understood that the embodiments shown in Figures 13A-13M are merely a set of examples. In the embodiments of this application, the electronic device 100 may also display more, fewer, or different interfaces than those in the above embodiments when creating gestures, and this application does not limit this. In some embodiments, the identification mark may also adopt a different display method than those in the above embodiments, such as using different display colors, or using different fonts or font sizes to represent actions in different identification states. This application does not limit the specific display method of the identification mark.

[0484] In this application embodiment, the electronic device 100 may use one or more methods such as display screen, voice broadcast, vibration, and indicator light flashing to prompt the user with information such as successful gesture creation or gesture creation failure. This application does not limit this.

[0485] Understandably, since the electronic device 100 recognizes gestures by recognizing one or more actions, by setting time intervals between multiple actions, gestures with different execution speeds can be obtained based on one or more actions and the time taken to execute them, such as fast gestures and slow gestures. In one possible implementation, the electronic device 100 can evaluate whether a gesture is fast or slow based on the time interval between two adjacent actions in a sequence of actions.

[0486] Accordingly, in one possible implementation, the electronic device 100 can display an option for "fast gesture" or "slow gesture," allowing the user to select the corresponding option to set the gesture as fast or slow. It is understood that the electronic device 100 can preset a threshold for the time interval between adjacent actions in a fast or slow gesture. For example, for a gesture consisting of a sequence of actions such as "clench fist - release - clench fist - release," the electronic device can identify the time interval between the second action "release" and the third action "clench fist." If the time interval is less than the preset threshold, the gesture is considered to be two fast clenched fists; if the time interval is greater than the preset threshold, the gesture is considered to be two slow clenched fists. It is understood that the above explanation uses the time interval between the second and third actions to evaluate whether a gesture is fast or slow as an example. It is also understood that the gesture can be evaluated based on all the time intervals between adjacent actions in the action sequence. For example, when the time intervals between all adjacent actions are less than the preset threshold, the gesture is considered fast; otherwise, it is considered slow. Alternatively, if the average time interval between any two adjacent actions is less than a preset threshold, the gesture is considered a fast gesture; otherwise, it is considered a slow gesture. This application does not limit the method for determining fast or slow gestures.

[0487] In another possible implementation, the user can set the time interval between each two adjacent actions in the gesture to set the gesture as a fast gesture or a slow gesture. The electronic device 100 determines whether the gesture is a fast gesture or a slow gesture by comparing the time interval between each adjacent action in the detected gesture with the time interval set by the user.

[0488] Understandably, by setting up fast and slow gestures, multiple gestures can be obtained based on the same sequence of actions, further expanding the number of gestures.

[0489] It is understood that the embodiments of this application use fast and slow gestures as examples for illustration. More settings can be made for the time interval between each two adjacent actions in the gesture, so as to obtain more gestures with different execution speeds, such as slow gestures, medium-speed gestures, and fast gestures. The embodiments of this application do not limit this.

[0490] In some application scenarios, the electronic device 100 can display a gesture editing interface with gesture D. The gesture editing interface can include adding operation controls. The electronic device 100 can receive and respond to the user's operation on adding operation controls, and display an operation selection interface. The operation selection interface can include one or more operation options, and each operation option can correspond to one operation. The electronic device 100 can receive and respond to the user's operation on one of the operation options, and determine that the operation corresponding to gesture D is the operation corresponding to that operation option.

[0491] In this way, the electronic device 100 can be set to perform operations corresponding to gestures to meet the needs of users in different scenarios.

[0492] Figures 14A-14D show schematic diagrams of a set of interface diagrams for setting gestures and corresponding operations provided in the embodiments of this application.

[0493] For example, in some embodiments, the electronic device 100 may display the gesture editing interface 1400 as shown in FIG14A after the gesture D is successfully created. In other embodiments, the electronic device 100 may also display the gesture D option in the gesture setting interface 1300 after the gesture D is successfully created. The electronic device 100 may receive and respond to the user's click operation on the gesture D option and display the gesture editing interface 1400 as shown in FIG14D.

[0494] As shown in Figure 14A, the gesture editing interface 1400 may include an edit gesture control 1401, a naming control 1402, and an add operation control 1403. The edit gesture control 1401 can be used to trigger the electronic device 100 to edit the action components of gesture D. The naming control 1402 can be used to trigger the electronic device 100 to change the name of gesture D. The add operation control 1403 can be used to trigger the electronic device 100 to set the operation corresponding to gesture D.

[0495] The electronic device 100 can receive and respond to the user's click operation on the added operation control 1403, and display the operation selection interface 1410 as shown in FIG14B.

[0496] As shown in Figure 14B, the operation selection interface 1410 may include one or more operation options, each operation option corresponding to one operation. For example, the unlock screen option 1411 corresponds to the operation of unlocking the screen; the lock screen option 1412 corresponds to the operation of locking the screen; and the payment code option 1413 corresponds to the operation of opening the payment code. In some embodiments, the operation selection interface 1410 may also include an input control, which can be used to determine the name of the operation entered by the user based on the user's input operation.

[0497] In some embodiments, the electronic device 100 may receive and respond to a user's click on the unlock screen option 1411, displaying an operation confirmation interface 1420 as shown in FIG14C.

[0498] As shown in Figure 14C, the operation confirmation interface 1420 may include an operation confirmation prompt 1421, which prompts the user to confirm whether to set the operation corresponding to gesture D as unlocking the screen. Furthermore, the operation confirmation interface 1420 may include a confirm control 1422 and a return control 1423. The confirm control 1422 can be used to trigger the electronic device 100 to confirm setting the operation corresponding to gesture D as unlocking the screen, and the return control 1423 can be used to trigger the electronic device 100 to redisplay the operation selection interface 1410 shown in Figure 14B.

[0499] In some embodiments, the electronic device 100 may receive and respond to a user's click operation on the determining control 1422, displaying a gesture editing interface 1400 for gesture D as shown in FIG14D. In other embodiments, the electronic device 100 may receive and respond to a user's click operation on the unlock screen option 1411 shown in FIG14B, displaying a gesture editing interface 1400 for gesture D as shown in FIG14D.

[0500] As shown in Figure 14D, the gesture editing interface 1400 can display an edit gesture control 1401, a naming control 1402, and an operation editing control 1404. The specific functions of the edit gesture control 1401 and the naming control 1402 can be found in the description of the embodiment shown in Figure 14A. The operation editing control 1404 can include the operation name corresponding to gesture D, such as unlocking the screen. Optionally, the operation editing control 1404 can also be used to trigger the electronic device 100 to modify or add the operation corresponding to gesture D.

[0501] It is understood that the embodiments shown in Figures 14A-14D are only a set of examples. In the embodiments of this application, the electronic device 100 may also display more, fewer, or different interfaces than the above embodiments when setting the operation corresponding to gesture D. This application does not limit this.

[0502] In some application scenarios, electronic device 100 can display an operation selection interface with gesture D. The operation selection interface includes one or more operation options, each corresponding to an operation. These one or more operation options may include operation option 1, which may correspond to a high-security-level operation (such as opening a payment code or other payment-related operation). Electronic device 100 can receive and respond to the user's operation on operation option 1 to determine whether the security level of gesture D is high-security. If gesture D is high-security, electronic device 100 can determine that the operation corresponding to gesture D is the operation corresponding to operation option 1; if gesture D is not high-security, electronic device 100 can display a security prompt interface, which can be used to inform the user that the security level of gesture D is insufficient. The security prompt interface may include an edit gesture control. Electronic device 100 can receive and respond to the user's operation on the edit gesture control, re-edit the action composition of gesture D, or display other gestures that meet the security level for the user to choose from.

[0503] In this way, the electronic device 100 can ensure that high-security operations correspond to high-security gestures, so as to avoid losses to users due to overly simple gestures and improve the security of operations involving user property.

[0504] Figures 15A-15G illustrate interface diagrams of a set of gestures that match the security level of the operation, as provided in the embodiments of this application.

[0505] As shown in Figure 15A, the electronic device 100 can display an operation selection interface 1410, which may include a payment code option 1413. The operation corresponding to the payment code option 1413 can be opening the payment code. Other specific content of the operation selection interface 1410 can be referred to the relevant description in the embodiment shown in Figure 14B above, and will not be repeated here.

[0506] Electronic device 100 can receive and respond to the user's click operation on payment code option 1413. After confirming that opening the payment code is a high-security operation, electronic device 100 can display a security prompt interface 1500 as shown in Figure 15B.

[0507] As shown in Figure 15B, the security prompt interface 1500 may include a security prompt 1501, an edit gesture control 1502, a continue control 1503, and a return control 1504. The security prompt 1501 can be used to inform the user that gesture D has low security and does not match the security level of the selected operation. In some embodiments, the security prompt 1501 may include text, such as "This gesture has low security, continue?". The edit gesture control 1502 can be used to trigger the electronic device 100 to re-edit the action components of gesture D. The continue control 1503 can be used to trigger the electronic device 100 to change the operation corresponding to gesture D to opening the payment code. The return control 1504 can be used to trigger the electronic device 100 to return to the operation selection interface 1410 shown in Figure 15A and reselect the operation corresponding to gesture D.

[0508] In some embodiments, the action library stored in the electronic device 100 may also include a security level for each action. The security level may include a high security level and a low security level; optionally, it may also include a medium security level, etc., which are not limited herein. The electronic device 100 can receive and respond to a user's click operation on the edit gesture control 1502, displaying a gesture creation interface 1510 as shown in FIG15C. The gesture creation interface 1510 may include one or more actions from the action library that match the security level of the user's selected operation. It should be noted that matching the security level of a gesture (or action) with the operation means that the security level of the gesture (or action) is higher than or equal to the security level of the operation.

[0509] As shown in Figure 15C, the gesture creation interface 1510 may include one or more action options, each of which may correspond to a high-security action. For example, the one or more action options may include a thumb swiping up on the side of the index finger (option 1511), a thumb swiping down on the side of the index finger (option 1512), and a thumb tapping on the second joint side of the index finger (option 1513).

[0510] Electronic device 100 can receive and respond to a user's tap operation on option 1511, which involves swiping the thumb up on the side of the index finger, and display gesture creation interface 1520 as shown in FIG15D.

[0511] As shown in Figure 15D, the gesture creation interface 1520 may include a selected action 1521 and one or more action options. The selected action 1521 may include a sequence of actions already selected by the user, such as "slide your thumb up along the side of your index finger". In some embodiments, each action option in the one or more action options may correspond to an action in an action library, and the action is sequential with the last action in the selected action 1521. Optionally, the action corresponding to the one or more action options may also be a high-security action in the action library. For example, the one or more action options may include the option 1522 of sliding your thumb down along the side of your index finger and the option 1523 of tapping your thumb on the second joint side of your index finger, etc. In addition, the gesture creation interface 1520 may also include a confirmation control 1524. The specific functional descriptions of the confirmation control 1524, the selected action 1521, and the one or more action options can be referred to the relevant descriptions in the embodiment shown in Figure 13C above, and will not be repeated here.

[0512] Electronic device 100 can receive and respond to a user's tap operation on the thumb swiping down the side of the index finger option 1522, and display the gesture creation interface 1530 as shown in FIG15E.

[0513] As shown in Figure 15E, the gesture creation interface 1530 may include a selected action 1531, a confirmation control 1533, and one or more action options. For example, one or more action options may include a thumb tapping option 1532 on the second joint side of the index finger. The selected action 1531 may include a series of actions selected by the user, and all actions in the selected action 1531 can be arranged according to the order selected by the user. For example, the selected actions 1531, in a top-to-bottom order, may include "thumb slides up on the side of the index finger" and "thumb slides down on the side of the index finger." Furthermore, other specific functions of the selected action 1531, the one or more action options, and the confirmation control 1533 can be found in the relevant descriptions in the embodiment shown in Figure 15D above, and will not be repeated here.

[0514] The electronic device 100 can receive and respond to the user's click operation on the confirmation control 1533 and display the gesture test interface 1540 as shown in FIG15F.

[0515] As shown in Figure 15F, the gesture testing interface 1540 can include all actions selected by the user. For example, the selected actions can include swiping the thumb up along the side of the index finger (1541) and swiping the thumb down along the side of the index finger (1542). The order of these two actions on the gesture testing interface 1540 is exactly the same as the order of the actions selected by the user. Furthermore, each action can also correspond to an identification mark; for example, swiping the thumb up along the side of the index finger (1541) corresponds to identification mark 1543, and swiping the thumb down along the side of the index finger (1542) corresponds to identification mark 1544. A detailed description of the function of the identification marks can be found in the relevant description in the embodiment shown in Figure 13F above.

[0516] The electronic device 100 can recognize the actions performed by the user. After recognizing that the user has performed all the actions in the gesture test interface 1540 in sequence, as shown in Figure 15F, the electronic device 100 can change the status of the recognition identifier 1543 and the recognition identifier 1544 to the recognition success status.

[0517] After all actions are successfully recognized, the electronic device 100 can display the operation confirmation interface 1550 as shown in Figure 15G.

[0518] As shown in Figure 15G, the operation confirmation interface 1550 may include an operation confirmation prompt 1551, which prompts the user to confirm whether to set the operation corresponding to gesture D as opening the payment code. In addition, the operation confirmation interface 1550 may also include a confirm control 1552 and a return control 1553. The confirm control 1552 can be used to trigger the electronic device 100 to confirm setting the operation corresponding to gesture D as opening the payment code, and the return control 1553 can be used to trigger the electronic device 100 to redisplay the operation selection interface 1410 shown in Figure 15A.

[0519] It is understood that the embodiments shown in Figures 15A-15G are merely a set of examples. In the embodiments of this application, the electronic device 100 may also display more, fewer, or different interfaces than the above embodiments during the process of editing gesture D, or display more, fewer, or different action options in the gesture creation interface than the above embodiments. Furthermore, the gestures created by the user and the operations selected by the user may also differ from the above embodiments, and this application does not limit them here.

[0520] In some application scenarios, when the security level of a user-selected operation is detected to be higher than the security level of a gesture, the electronic device 100 can prompt the user to add more actions to the gesture. This way, the security level of the gesture can be improved by increasing the number of actions, so that the security level of the gesture matches the security level of the operation.

[0521] Figures 16A-16F show a set of interface diagrams provided by embodiments of this application for increasing the number of actions in a gesture when the gesture safety level is lower than the operation safety level.

[0522] As shown in Figure 16A, the electronic device 100 can display an operation selection interface 1410. The operation selection interface 1410 may include a payment code option 1413, which corresponds to the operation of opening the payment code. Other specific contents of the operation selection interface 1410 can be referred to the relevant description in the embodiment shown in Figure 14B above, and will not be repeated here.

[0523] The electronic device 100 can receive and respond to the user's click operation on the payment code option 1413. When it is determined that the security level of gesture D is lower than the security level of opening the payment code operation, a security prompt interface 1600 as shown in Figure 16B is displayed.

[0524] As shown in Figure 16B, the safety prompt interface 1600 may include a safety prompt 1601, which is used to indicate to the user that the number of actions in gesture D is insufficient, and also to suggest adding more actions to gesture D. In some embodiments, the safety prompt 1601 may include text, such as "This gesture includes few actions; it is recommended to increase to 4 actions." The safety prompt interface 1600 may also include an add action control 1602, a continue control 1603, and a return control 1604. The add action control 1602 can be used to trigger the electronic device 100 to display one or more action options, so as to add actions based on the user's selection of the existing gesture D. The functional descriptions of the continue control 1603 and the return control 1604 can be referred to the relevant descriptions in the embodiment shown in Figure 15B above, and will not be repeated here.

[0525] The electronic device 100 can receive and respond to the user's click operation on the added action control 1602, and display the gesture creation interface 1610 as shown in FIG16C.

[0526] As shown in Figure 16C, the gesture creation interface 1610 may include a selected action 1611 and one or more action options, such as a fist option 1612 and a pinch option 1613. Furthermore, the gesture creation interface 1610 may also include a confirmation control 1614. The selected action 1611 includes all actions in gesture D before the user clicks the add action control 1602, and the order of all actions in the selected action 1611 is the same as the order of actions in gesture D. The confirmation control 1614 can be used to trigger the electronic device 100 to determine the action string in the selected action 1611 as the modified gesture D.

[0527] The electronic device 100 can receive and respond to a user's tap operation on the pinch option 1613, and display the gesture creation interface 1620 as shown in FIG16D.

[0528] As shown in Figure 16D, the gesture creation interface 1620 may include a selected action 1621 and one or more action options, such as a release option 1622. The gesture creation interface 1620 may also include a confirmation control 1623. The selected action 1621 may include all the actions in the selected action 1611 shown in Figure 16C, and also includes a newly selected pinch by the user.

[0529] The electronic device 100 can receive and respond to a user's click operation on the confirmation control 1623, and the electronic device 100 can display the gesture test interface 1630 as shown in FIG16E.

[0530] As shown in Figure 16E, the gesture testing interface 1630 can include all actions in the modified gesture D, such as clenching a fist, swiping down with a knuckle, releasing, and pinching. It should be noted that the order of all actions is the same as the order of all actions in gesture D. Furthermore, each action can correspond to a recognition identifier; for example, clenching a fist corresponds to identification identifier 1631, swiping down with a knuckle corresponds to identification identifier 1632, releasing corresponds to identification identifier 1633, and pinching corresponds to identification identifier 1634. A detailed description of the function of the identification identifiers can be found in the relevant description in the embodiment shown in Figure 13F above.

[0531] After the electronic device 100 detects that the user sequentially performs a fist-knuckle-release-pinch, in some embodiments, the electronic device 100 can change the status of all recognition markers to recognition success in the gesture test interface 1630 shown in Figure 16E.

[0532] After confirming that all actions in the modified gesture D have been successfully recognized in sequence, the electronic device 100 can display the operation confirmation interface 1640 as shown in Figure 16F.

[0533] As shown in Figure 16F, the operation confirmation interface 1640 may include an operation confirmation prompt 1641, a confirm control 1642, and a return control 1643. The operation confirmation prompt 1641 prompts the user to confirm whether to set the operation corresponding to gesture D as opening the payment code. The confirm control 1642 triggers the electronic device 100 to confirm setting the operation corresponding to gesture D as opening the payment code. The return control 1643 triggers the electronic device 100 to redisplay the operation selection interface 1410 shown in Figure 16A.

[0534] Electronic device 100 can receive and respond to the user's click operation on the control 1642, and set the operation corresponding to gesture D to open the payment code.

[0535] It is understood that the embodiments shown in Figures 16A-16F are only a set of examples. In the embodiments of this application, the electronic device 100 may also add more, fewer or different actions than the above embodiments when editing gesture D, and gesture D may also be a gesture different from the above embodiments. The operation selected by the user may also be an operation different from the above embodiments. This application does not limit this.

[0536] In some application scenarios, when the security level of the user-selected operation is detected to be higher than the security level of the gesture, the electronic device 100 can prompt the user to add one or more high-security-level actions to the gesture. This way, the security level of the gesture can be improved by adding high-security-level actions, so that the security level of the gesture matches the security level of the operation.

[0537] Figures 16G-16K show a set of interface diagrams provided by embodiments of this application, which add high-security-level actions in gestures when the gesture security level is lower than the operation security level.

[0538] For example, electronic device 100 can receive and respond to a user's click operation on the payment code option 1413 shown in FIG16A, and display a security prompt interface 1650 as shown in FIG16G when it is determined that the security level of gesture D is lower than the security level of opening the payment code operation.

[0539] As shown in Figure 16G, the security prompt interface 1650 may include a security prompt 1651, which is used to inform the user that gesture D has a low security level and to suggest adding a high-security-level action to gesture D. In some embodiments, the security prompt 1651 may include text, such as "This gesture has low security; it is recommended to add a high-security-level action." The security prompt interface 1650 may also include an add action control 1652, a continue control 1653, and a return control 1654. The add action control 1652 can be used to trigger the electronic device 100 to display action options corresponding to one or more high-security-level actions, so as to add a high-security-level action based on the user's selection to the existing gesture D. The functional descriptions of the continue control 1653 and the return control 1654 can be found in the relevant descriptions in the embodiment shown in Figure 15B above, and will not be repeated here.

[0540] The electronic device 100 can receive and respond to the user's click operation on the added action control 1652, and display the gesture creation interface 1660 as shown in FIG16H.

[0541] As shown in Figure 16H, the gesture creation interface 1660 may include a selected action 1661 and one or more action options. For example, the one or more action options may include an option 1662 for the thumb to slide up along the side of the index finger and an option 1663 for the thumb to slide down along the side of the index finger. Furthermore, the gesture creation interface 1660 may also include a confirmation control 1664. The selected action 1661 includes all actions in gesture D before the user clicks the add action control 1652, and the order of all actions in the selected action 1661 is the same as the order of actions in gesture D. The confirmation control 1664 can be used to trigger the electronic device 100 to determine the action string in the selected action 1661 as the modified gesture D.

[0542] Electronic device 100 can receive and respond to a user's tap operation on option 1662, which involves swiping the thumb up on the side of the index finger, and display gesture creation interface 1670 as shown in FIG16I.

[0543] As shown in Figure 16I, the gesture creation interface 1670 may include a selected action 1671 and one or more action options. For example, the one or more action options may include the option 1672 of sliding the thumb down along the side of the index finger. The gesture creation interface 1670 may also include a confirmation control 1673. The selected action 1671 may include all the actions in the selected action 1661 shown in Figure 16H, and also includes the user's newly selected action of sliding the thumb up along the side of the index finger.

[0544] Electronic device 100 can receive and respond to user click operation on confirmation control 1673, and electronic device 100 can display gesture test interface 1680 as shown in FIG16J.

[0545] As shown in Figure 16J, the gesture testing interface 1680 can include all actions in the modified gesture D, such as clenching a fist, swiping down with the knuckles, releasing, and sliding the thumb up along the side of the index finger. It should be noted that the order of all these actions is the same as the order of all actions in gesture D. Furthermore, each action can correspond to a recognition identifier; for example, clenching a fist corresponds to identification identifier 1681, swiping down with the knuckles corresponds to identification identifier 1682, releasing corresponds to identification identifier 1683, and pinching corresponds to identification identifier 1684. A detailed description of the function of the identification identifiers can be found in the relevant description in the embodiment shown in Figure 13F above.

[0546] After the electronic device 100 detects that the user sequentially performs a fist-knuckle swipe-release-thumb swipe-index finger side, in some embodiments, the electronic device 100 can change the status of all recognition markers to recognition success in the gesture test interface 1680 shown in Figure 16J.

[0547] After confirming that all actions in the modified gesture D have been successfully recognized in sequence, the electronic device 100 can display an operation confirmation interface 1690 as shown in Figure 16K.

[0548] As shown in Figure 16K, the operation confirmation interface 1690 may include an operation confirmation prompt 1691, a confirm control 1692, and a return control 1693. The operation confirmation prompt 1691 prompts the user to confirm whether to set the operation corresponding to gesture D as opening the payment code. The confirm control 1692 triggers the electronic device 100 to confirm setting the operation corresponding to gesture D as opening the payment code. The return control 1693 triggers the electronic device 100 to redisplay the operation selection interface 1410 shown in Figure 16A.

[0549] Electronic device 100 can receive and respond to the user's click operation on the confirmation control 1692, and set the operation corresponding to gesture D to open the payment code.

[0550] It is understood that the embodiments shown in Figures 16G-16K are only a set of examples. In the embodiments of this application, the electronic device 100 may also add more, fewer, or different actions than the above embodiments when editing gesture D, and gesture D may also be a gesture different from the above embodiments. The operation selected by the user may also be an operation different from the above embodiments. This application does not limit these actions.

[0551] In some application scenarios, electronic device 100 can first select an operation, and then select the corresponding gesture from the pre-created gestures. Alternatively, electronic device 100 can also create the corresponding gesture after selecting an operation. In this way, electronic device 100 can select or create appropriate gestures based on the security level of the operation, so that the security level of the gesture matches the security level of the operation.

[0552] Figures 17A-17D show schematic diagrams of the interface corresponding to a set of selection operations provided in the embodiments of this application.

[0553] As shown in Figure 17A, the electronic device 100 may display an operation settings interface 1700, which may include one or more operation options, such as unlock screen option 1701, lock screen option 1702, and payment code option 1703. Each operation option may correspond to one operation. Each operation option can be used to trigger the electronic device 100 to set the gesture corresponding to that operation.

[0554] Electronic device 100 can receive and respond to a user's click operation on payment code option 1703, and display gesture selection interface 1710 as shown in FIG17B.

[0555] As shown in Figure 17B, the gesture selection interface 1710 may include one or more gesture options, such as gesture A, gesture B, gesture D, etc. Each gesture option can correspond to one gesture. The gesture corresponding to the one or more gesture options can be a created gesture or a gesture set at the factory of the electronic device 100. In addition, the gesture option interface 1710 may also include a gesture creation control 1712. The gesture creation control 1712 can be used to trigger the electronic device 100 to create a gesture corresponding to the user's selected operation.

[0556] The electronic device 100 can receive and respond to the user's tap operation on gesture option B 1711, and display the operation confirmation interface 1720 as shown in FIG17C.

[0557] As shown in Figure 17C, the operation confirmation interface 1720 may include an operation confirmation prompt 1721, a confirm control 1722, and a return control 1723. The operation confirmation prompt 1721 prompts the user to confirm whether to set the gesture corresponding to opening the payment code as gesture B. The confirm control 1722 triggers the electronic device 100 to confirm setting the gesture corresponding to opening the payment code as gesture B. The return control 1723 triggers the electronic device 100 to return to the gesture selection interface 1710 shown in Figure 17B.

[0558] The electronic device 100 can receive and respond to the user's click operation on the confirmation control 1722 and display the payment code setting interface 1730 as shown in Figure 17D.

[0559] As shown in Figure 17D, the payment code settings interface 1730 may include a selected gesture 1731, which can be used to indicate the gesture corresponding to opening the payment code, namely gesture B (clenching a fist - releasing - sliding the thumb up along the side of the index finger). The payment code settings interface 1730 may also include a gesture change control 1732. The gesture change control 1732 can be used to trigger the electronic device 100 to change the gesture corresponding to opening the payment code.

[0560] It is understood that the embodiments shown in Figures 17A-17D are merely illustrative examples illustrating that an operation can be selected before selecting or creating a gesture. In the embodiments of this application, after selecting an operation, the electronic device 100 may also select a gesture different from the above embodiments, or create a gesture different from the above embodiments. This application does not limit this.

[0561] The functional modules of an electronic device 100 provided in the embodiments of this application are described below.

[0562] Figure 18 shows a schematic diagram of the functional modules of an electronic device 100 provided in an embodiment of this application.

[0563] As shown in Figure 18, the electronic device 100 may include an action recognition module 1801, a gesture recognition module 1802, a gesture-operation module 1803, an operation execution module 1805, a user interaction module 1806, etc. Optionally, the electronic device 100 may also include a historical data module 1804 and a communication module 1807. Wherein:

[0564] The action recognition module 1801 can store an action library, which may include multiple actions and corresponding features for each action. When a feature in the action library is detected, the action recognition module 1801 can identify the action performed by the user based on the correspondence between the feature and the action. Specifically, the action recognition module 1801 can acquire signals collected by devices (such as PPGs or accelerometers). The action recognition module 1801 can also preprocess the acquired signals to eliminate useless signal components and adjust the numerical range of the signal data to avoid noise and differences in numerical range affecting the gesture recognition results. For example, a high-pass filter can be used to filter out the DC component of the signal while retaining the AC component; z-score normalization can be used to adjust the numerical range of the signal data. The action recognition module 1801 can detect action features in the signal from the data buffer using methods such as machine learning, deep learning, and empirical rules, and input these features into a pre-trained gesture classifier, avoiding sending useless data into the model inference and reducing the power consumption of the model inference. The action recognition module 1801 can obtain the action or no-action (the user did not perform an action) corresponding to the feature based on the gesture classifier. The gesture classifier can be a traditional machine learning model, such as SVM or decision tree, or a deep learning network. If the gesture classifier outputs an empty action (i.e., no user action is detected) within a preset time (such as the duration of the timer described in S402), the gesture recognition ends. If the gesture classifier recognizes an action, it can record the action in the result sequence and repeat the above process until the gesture recognition ends.

[0565] After recognizing that the user has performed an action from the action library, the action recognition module 1801 can send the recognized action to the gesture recognition module 1802. In some embodiments, the action recognition module 1801 can send a single action to the gesture recognition module 1802 after recognizing it. In other embodiments, the action recognition module 1801 can send a sequence of recognized actions to the gesture recognition module 1802 after recognizing multiple actions. In some embodiments, the action recognition module 1801 can send the recognized actions to the historical data module 1804.

[0566] The gesture recognition module 1802 can store one or more gestures. In some embodiments, the gesture recognition module 1802 can also receive custom gestures sent by the user interaction module 1806 and store newly created custom gestures by the user. The gesture recognition module 1802 can receive one or more actions (or action strings) sent by the action recognition module 1801 and determine the gesture performed by the user based on the received one or more actions (or action strings). After determining the gesture performed by the user, the gesture recognition module 1802 can send the recognized gesture to the gesture-operation module 1803. In some embodiments, the gesture recognition module 1802 can also send the recognized gesture to the historical data module 1804.

[0567] The gesture-operation module 1803 can store the correspondence between gestures and operations. The gesture-operation module 1803 can receive gestures sent by the gesture recognition module 1802 and determine the operation corresponding to the gesture based on the correspondence between gestures and operations. In some embodiments, the gesture-operation module 1803 can send the operation to the operation execution module 1805. In some embodiments, if the gesture-operation module 1803 determines that the gesture corresponds to an operation performed by another electronic device, then the gesture-operation module 1803 can send the operation performed by the other electronic device to the communication module 1807.

[0568] The historical data module 1804 can receive one or more actions sent by the action recognition module 1801 and store the recognition record of each action. In some embodiments, the historical data module 1804 can also receive one or more gestures sent by the gesture recognition module 1802 and store the recognition record of each gesture. In some embodiments, the historical data module 1804 can also interact with the user interaction module 1806, and based on the stored action recognition records and / or gesture recognition records, send actions and / or gestures that have been successfully recognized a lot (or have a high recognition success rate) to the user interaction module 1806.

[0569] The operation execution module 1805 can receive operations sent by the gesture-operation module 1803 and execute those operations.

[0570] The user interaction module 1806 can interact with the user, such as receiving user operations and outputting corresponding results (e.g., gesture operation prompts, action options, etc.). In some embodiments, the user interaction module 1806 can interact with the historical data module 1804 during the user's gesture execution or gesture creation process to obtain actions or gestures with a high number of successful recognitions or a high recognition success rate, and recommend gestures or actions in the gesture operation prompts based on the recognition success rate. Specifically, it can sort the confidence scores of each recognized gesture or action according to the historical output of the gesture classifier, and recommend gestures or actions with high confidence scores.

[0571] In some embodiments, the communication module 1807 can receive operations performed by other electronic devices sent by the gesture-operation module 1803, and send instructions to the other electronic devices, which are used to instruct the receiver to perform the operations performed by the other electronic devices.

[0572] It is understood that the embodiment shown in FIG18 is only an example. In the embodiments of this application, the electronic device 100 may include more, fewer, or different modules than those shown in FIG18, and any of the above modules may be combined into one module, or any of the above modules may be split into multiple modules. This application does not limit this.

[0573] The functional modules of a gesture control system 10 provided in the embodiments of this application are described below.

[0574] Figure 19 shows a schematic diagram of the functional modules of a gesture control system 10 provided in an embodiment of this application.

[0575] As shown in Figure 19, the gesture control system 10 may include electronic device 100 and electronic device 200. Electronic device 100 may include a motion recognition module 1801, a gesture recognition module 1802, a gesture-operation module 1803, a user interaction module 1806, and a communication module 1807. Optionally, it may also include a historical data module 1804 and an operation execution module 1805. Electronic device 200 may include a communication module 1901 and an operation execution module 1902.

[0576] In the electronic device 100, the specific functions of each module can be referred to the relevant descriptions of the corresponding modules in the embodiment shown in Figure 18 above, and will not be repeated here.

[0577] In electronic device 200, communication module 1901 can receive instructions sent by electronic device 100 and send the instructions to operation execution module 1902.

[0578] The operation execution module 1902 can receive instructions sent by the communication module 1901 and execute the operation specified by the instructions.

[0579] It is understood that the embodiment shown in Figure 19 above is only an example. In the embodiments of this application, the gesture control system 10 may also include more or different electronic devices than the above embodiments, and each electronic device may include more, fewer or different modules than the above embodiments, and any of the above modules may be combined into one module, or any of the above modules may be split into multiple modules. This application does not limit this.

[0580] For ease of subsequent description, the electronic devices 100 and 200 described above can be collectively referred to as a device. It should be understood that the division of units within this device is merely a logical functional division; in actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, the units within the device can be implemented by a processor calling software; for example, the device includes a processor connected to a memory containing instructions. The processor calls the instructions stored in the memory to implement any of the above methods or to realize the functions of each unit within the device. The processor can be, for example, a general-purpose processor, such as a central processing unit (CPU) or a microprocessor, and the memory can be internal or external to the device. Alternatively, the units in the device can be implemented as hardware circuits. The functionality of some or all units can be achieved through the design of these hardware circuits, which can be understood as one or more processors. For example, in one implementation, the hardware circuit is an application-specific integrated circuit (ASIC). The functionality of some or all of the above units is achieved through the design of the logical relationships between the components within the circuit. In another implementation, the hardware circuit can be implemented using a programmable logic device (PLD). Taking a field-programmable gate array (FPGA) as an example, it can include a large number of logic gates. The connection relationships between the logic gates are configured through a configuration file, thereby achieving the functionality of some or all of the above units. All units of the above device can be implemented entirely through processor-invoked software, entirely through hardware circuits, or partially through processor-invoked software with the remaining parts implemented through hardware circuits.

[0581] In this application embodiment, a processor is a circuit with data processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a CPU, microprocessor, graphics processing unit (GPU) (which can be understood as a type of microprocessor), or digital signal processor (DSP). In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships of hardware circuits are fixed or reconfigurable. For example, the processor is a hardware circuit implemented as an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. Furthermore, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a Neural Network Processing Unit (NPU), Tensor Processing Unit (TPU), or Deep Learning Processing Unit (DPU).

[0582] As can be seen, each unit in the above device can be one or more processors (or processing circuits) configured to implement the above methods, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0583] Furthermore, the units in the above devices can be integrated in whole or in part, or they can be implemented independently. In one implementation, these units are integrated together as a system-on-a-chip (SOC). The SOC may include at least one processor for implementing any of the above methods or implementing the functions of the units in the device. The at least one processor may be of different types, such as CPU and FPGA, CPU and artificial intelligence processor, CPU and GPU, etc.

[0584] The following describes a possible physical structure of the electronic device 400 provided in the embodiments of this application.

[0585] For example, Figure 20 shows a schematic diagram of the physical structure of an electronic device 400 provided in an embodiment of this application.

[0586] As shown in Figure 20, the electronic device 400 may include a processor 2001, a memory 2002, and a bus 2005. Optionally, it may also include a transmitter 2003 and a receiver 2004. The processor 2001, memory 2002, transmitter 2003, and receiver 2004 may be interconnected or interconnected via the bus 2005.

[0587] For example, memory 2002 is used to store computer programs and data of electronic device 400. Memory 2002 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM).

[0588] The software or program code required for all or part of the functions of the electronic device 400 in the above method embodiments is stored in the memory 2002.

[0589] In one possible implementation, if the software or program code required for some functions is stored in the memory 2002, then in addition to calling the program code in the memory 2002 to implement some functions, the processor 2001 can also cooperate with other components (such as the transmitter 2003 and the receiver 2004) to jointly complete other functions described in the method embodiment (such as the function of receiving or sending data).

[0590] Transmitter 2003 and receiver 2004 are used to support electronic device 400 in communication, such as receiving or sending data or signals.

[0591] For example, processor 2001 may be a CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor types, as described above. Processor 2001 may be used to read programs stored in memory 2002 and execute operations performed by electronic device 400 in any of the above embodiments.

[0592] The specific operation and beneficial effects of each unit in the electronic device 400 shown in Figure 20 can be found in the corresponding descriptions in the above method embodiments, and will not be repeated here. It should be noted that the electronic device 400 can be either the electronic device 100 or the electronic device 200 in the above embodiments.

[0593] It is understood that the embodiment shown in FIG20 is only an example. In the embodiments of this application, the electronic device 400 may also include more, fewer, or different devices than those shown in FIG20. This application does not limit the scope of the invention.

[0594] The present application provides a gesture control method.

[0595] Figure 21 shows a flowchart of a gesture control method provided in an embodiment of this application.

[0596] As shown in Figure 21, the specific process of a gesture control method may include the following steps:

[0597] S2101. The first electronic device detects the user's first action.

[0598] The first electronic device has multiple actions pre-stored, which may include a first action, a second action, etc.

[0599] The first electronic device may be the electronic device 100 in the above embodiments.

[0600] For example, the first action can be action 1 in the embodiment shown in Figure 3 above.

[0601] S2102. The first electronic device detects the user's second action.

[0602] The second action can be an action performed by the user after the first action, such as the next action after the first action, or an action that is not adjacent to the first action after the first action, etc. This application does not limit it here.

[0603] S2103. The first electronic device detects the user's end gesture.

[0604] The method by which the first electronic device determines whether the user has ended the gesture can be referred to the relevant description in the embodiment shown in Figure 4 above.

[0605] S2104. The first electronic device executes a first operation corresponding to the first gesture, the first gesture including a first action and a second action, and the starting action of the first gesture is the first action.

[0606] For example, the first gesture can be gesture 1 in the embodiment shown in Figure 3 above, and the first operation can be operation 1 in the embodiment shown in Figure 3 above.

[0607] For example, the first gesture can also be gesture 2 in the embodiment shown in FIG5 above, and the first operation can be sending instruction 1 to electronic device 200 in the embodiment shown in FIG5.

[0608] It should be noted that in this application, a gesture may include one or more actions, that is, an action is used as the basic unit to construct a complete gesture.

[0609] In this way, gesture recognition can be achieved based on action recognition. An infinite number of gestures can be constructed from a finite number of actions, which can meet the diverse functional needs of users in different scenarios. In addition, since a gesture can be composed of many actions, gesture recognition based on action recognition is more lenient in terms of the time required for users to execute gestures compared to direct gesture recognition. Users can execute gestures more leisurely without worrying about gesture recognition failure due to slow execution.

[0610] In one possible implementation, the ending action of the first gesture is the second action; detecting the user's ending gesture specifically includes: no user action is detected within a first time period, the start time of the first time period is the time when the second action is detected, and the duration of the first time period is the first duration.

[0611] For example, the first duration can be duration 1 in the embodiment shown in Figure 4 above.

[0612] In this way, after an action is performed, as long as the user performs the next action within a preset time (e.g., 500 milliseconds, 300 milliseconds, etc.), it can be determined that the two actions performed by the user belong to the same gesture, thus realizing the continuity detection of gestures.

[0613] In one possible implementation, detecting the user's end gesture specifically includes: determining that among one or more gestures stored in the first electronic device, there is no gesture that includes a first action and a second action, whose starting action is the first action, and whose number of actions is greater than the number of actions already detected.

[0614] In this way, the electronic device can determine that the user's gesture has ended when there is no gesture in the pre-stored gestures that has more actions than the currently executed gesture, and the action sequence composed of the previous one or more actions is completely consistent with the currently executed gesture.

[0615] In one possible implementation, after detecting that the user has performed a first action, the method further includes: outputting a first prompt, the first prompt including one or more action guides, each action guide including one or more actions, the one or more actions being actions that the user can perform after the first action.

[0616] For example, the first prompt can be the gesture operation prompt 1 shown in Figure 3 above.

[0617] For example, the action guidance in the first prompt can be the optional action 1112 in the embodiment shown in FIG11C, the optional action string 1142 in the embodiment shown in FIG11K, or the gesture 1212, gesture 1213 or gesture 1214 in the embodiment shown in FIG12B.

[0618] In this way, during the user's gesture execution, a first prompt can be output based on the sequence of actions the user has already performed, indicating the actions the user can perform next.

[0619] In one possible implementation, one or more action guides include a first action guide, which in turn includes a second action.

[0620] For example, the first action guide can be the optional action 1112 in the embodiment shown in FIG11C above. In this case, the first action can be clenching a fist and the second action can be releasing it.

[0621] For example, the first action guide can also be the optional action sequence 1142 in the embodiment shown in FIG11K. In this case, the first action can be clenching a fist and the second action can be releasing it.

[0622] As another example, the first action guide can also be the gesture 1212 in the embodiment shown in Figure 12B. In this case, the first action can be clenching a fist, and the second action can be holding it.

[0623] In this way, the first action can guide the user and prompt them to perform the second action after the first action.

[0624] In one possible implementation, one or more action guides may also include a third action guide, which in turn includes a fourth action.

[0625] For example, the third action guide may be gesture 1213 or gesture 1214 in the embodiment shown in FIG12B.

[0626] In one possible implementation, the third action is guided after the first action, and the historical execution count of the fourth action is lower than the historical execution count of the second action.

[0627] It should be noted that the first electronic device can determine the order of the one or more action guides based on the historical execution count of the first action in the action guide, for example, displaying the action guide corresponding to the action with the highest historical execution count at the beginning. Alternatively, the first electronic device can also determine the order of the one or more action guides based on the historical execution count of the gesture corresponding to the action guide, for example, displaying the action guide corresponding to the gesture with the highest historical execution count at the beginning. Here, a gesture corresponding to an action guide refers to the gesture that the action guide instructs the user to perform.

[0628] In this way, the display order of one or more actions can be determined based on the historical number of times the actions have been executed.

[0629] In one possible implementation, one or more action guides may also include a fourth action guide, with the first action guide appearing before the fourth action guide, and the first action guide being more relevant to the first scene than the fourth action guide is to the first scene.

[0630] The relevance of the action guidance to the scene refers to the relevance of the gesture instructed by the action guidance to the scene. A description of the relevance of the gesture to the scene can be found in the relevant content following the embodiment shown in Figure 12E above, and will not be repeated here.

[0631] In this way, the display order of one or more action guides can be determined based on the relevance of the action guides to the scene.

[0632] In one possible implementation, the first action guidance also includes a name for the first operation, which indicates the operation triggered by the user after performing the first gesture.

[0633] For example, if the first action is guided as the optional action string 1142 in the embodiment shown in Figure 11K, then the name of the first operation can be "answer".

[0634] In this way, the user can be guided by the first action and prompted to perform the second action after the first action, which can trigger the first electronic device to perform the first operation.

[0635] In one possible implementation, the first prompt also includes a first action string, which is used to indicate an action performed by the user, and the first action string includes a first action.

[0636] For example, the first action sequence can be the executed action 1111 in the embodiment shown in FIG11C above, or the executed action 1141 in the embodiment shown in FIG11K.

[0637] In this way, the user can also be prompted with the first action string in the first prompt, i.e. the first action.

[0638] In one possible implementation, the first gesture also includes a third action; after detecting that the user has performed the first action, the method further includes: detecting that the user has performed the third action.

[0639] For example, if the first action is the optional action sequence 1142 in the embodiment shown in Figure 11K, then the third action can be clenching a fist or loosening it.

[0640] Thus, the first gesture can include three or more actions.

[0641] In one possible implementation, outputting a first prompt specifically includes: displaying a first action string based on a first display method; and displaying one or more action guides based on a second display method, wherein the first display method and the second display method are different.

[0642] The first display mode can be used to indicate that the first action sequence is an executed action, and the second display mode can be used to indicate that the action guide is a subsequent action that can be executed. In one possible implementation, the first electronic device can distinguish between the first and second display modes by different display modes such as display color, font, font size, and weight. In another possible implementation, the first electronic device can also distinguish between the first and second display modes by different symbols.

[0643] In this way, different display methods can be used to prompt the user about the actions that have been performed and the actions that can be performed next.

[0644] In one possible implementation, after detecting the user's second action, the method further includes: outputting a second prompt, the second prompt being used to prompt the user that ending the gesture can trigger the first electronic device to perform the first operation.

[0645] For example, the second prompt may be the end prompt 1119 in the embodiment shown in FIG11F above.

[0646] In this way, when a user's gesture can trigger a specific operation, the user can be prompted to end the gesture in order to trigger that specific operation.

[0647] In one possible implementation, the first action guide further includes a third action, which is displayed after the second action; after detecting the user's second action, the method further includes: outputting a third prompt; the third prompt includes the second action guide, which includes the third action; and / or, the third prompt further includes a second action string, which indicates an action performed by the user, and the second action string includes the first action and the second action.

[0648] For example, the second action guide may be the optional action string 1144 in the embodiment shown in FIG11L above, and the second action string may be the executed action 1143 in the embodiment shown in FIG11L.

[0649] In this way, after the user performs the second action, a third prompt can be output. The third prompt can be used to inform the user of changes in the action that has been performed, and / or to inform the user of subsequent actions that can be performed.

[0650] In one possible implementation, before detecting the user's first action, the method further includes: establishing a first communication connection with a second electronic device; and performing a first operation corresponding to the first gesture, specifically including: sending a first instruction to the second electronic device through the first communication connection, the first instruction being used to instruct the second electronic device to perform a second operation.

[0651] For example, the second electronic device may be the electronic device 200 in the above embodiments, and the second operation may be operation 2 in the embodiment shown in FIG5.

[0652] In this way, a first gesture can trigger a first electronic device to control a second electronic device to perform a second operation.

[0653] In one possible implementation, the method further includes: obtaining a first device type of the second electronic device based on the first communication connection; and sending a first instruction to the second electronic device through the first communication connection, specifically including: sending a first instruction to the second electronic device based on the first device type through the first communication connection.

[0654] For example, the first instruction may be instruction 1 in the embodiment shown in FIG5 above.

[0655] In this way, it is possible to determine whether the second operation is performed by controlling the second electronic device through the first electronic device, based on the device type of the second electronic device and the first gesture performed by the user.

[0656] In one possible implementation, before detecting the user's first action, the method further includes: displaying a first interface, the first interface including a first control; receiving and responding to the user's operation on the first control, displaying a second interface, the second interface including a first action option, the first action option corresponding to the first action; receiving and responding to the user's operation on the first action option, displaying one or more action options, the first or more action options including a second action option, the second action option corresponding to the second action; receiving and responding to the user's operation on the second action option, determining that the first gesture includes a first action and a second action, and that the first action is the starting action of the first gesture.

[0657] For example, the first interface can be the gesture setting interface 1300 in the embodiment shown in Figure 13A, and the first control can be the add gesture control 1301 in the embodiment shown in Figure 13A. The second interface can be the gesture creation interface 1310 shown in Figure 13B, and the first action option can be the fist clenching option 1311 shown in Figure 13B. The second action option can be the knuckle swipe down option 1322 in the embodiment shown in Figure 13C.

[0658] In this way, the first electronic device can create custom gestures based on the user's selected action sequence, making it easier for users to create gestures according to their own needs and enriching the diversity of gestures.

[0659] In one possible implementation, the second interface further includes a second action option and a third action option, the third action option corresponding to a fourth action, and there is no continuity between the fourth action and the first action; receiving and responding to the user's operation on the first action option, displaying one or more action options, specifically including: receiving and responding to the user's operation on the first action option, displaying the second action option, and not displaying the third action option.

[0660] In this way, during the user's gesture creation process, the first electronic device can display action options that are sequential with the last selected action. Since the fourth action corresponding to the third action option is not sequential with the first action, the first electronic device may not display the third action option after the user selects the first action.

[0661] In one possible implementation, after determining that the first gesture includes a first action and a second action, the method further includes: outputting a test prompt to prompt the user to perform the first gesture; detecting that the user has performed the first gesture; and outputting a success message to prompt the user that the first gesture has been successfully created.

[0662] For example, the test prompt may be text in the gesture test interface 1350 of the embodiment shown in Figure 13F above, such as "Please perform the following action", etc.

[0663] In this way, after the gesture is created, the user can be prompted to test whether the gesture can be successfully recognized, thereby improving the success rate of subsequent gesture recognition.

[0664] In one possible implementation, outputting test prompts specifically includes: displaying a test interface, the test interface including a first identification identifier corresponding to the first action and a second identification identifier corresponding to the second action, both the first identification identifier and the second identification identifier being in a pending identification state; detecting that the user performs a first gesture specifically includes: detecting that the user performs a first action; the first identification identifier changing from a pending identification state to a successful identification state; detecting that the user performs a second action; the second identification identifier changing from a pending identification state to a successful identification state.

[0665] For example, the test interface can be the gesture test interface 1350 shown in Figure 13F. The first identification identifier can be the identification identifier 1354 shown in Figure 13F, and the second identification identifier can be the identification identifier 1355 shown in Figure 13F. The state of the identification identifier can be referred to the relevant description in the embodiments shown in Figures 13F-13I above, and will not be repeated here.

[0666] In this way, during the gesture testing process, the user can be prompted with the recognition status of each action in the gesture, so that the user can keep track of the gesture test status in real time.

[0667] In one possible implementation, before detecting that the user has performed the first gesture, the method further includes: after outputting a test prompt, detecting a test failure; outputting a test failure prompt, which is used to inform the user that the first gesture has failed the test; receiving and responding to the user's operation to retest the first gesture, displaying a test interface, in which both the first identification identifier and the second identification identifier are in a state of waiting to be identified.

[0668] For example, a test failure message could be failure message 1371 as shown in Figure 13L above.

[0669] This way, if the gesture test fails, you can restart the gesture test.

[0670] In one possible implementation, after outputting a test failure message, the system receives and responds to the user's operation to retest the first gesture, and displays the test interface. Specifically, this includes receiving and responding to the user's operation to retest the first gesture, and if the number of times the first gesture test fails is less than a preset threshold, displaying the test interface.

[0671] In this way, if a gesture test fails, it can be determined whether to retest the gesture based on the number of failures.

[0672] In one possible implementation, if the number of failures is greater than or equal to a preset threshold, the first electronic device may also prompt the user to re-edit the first gesture.

[0673] In one possible implementation, after determining the first gesture, the method further includes: displaying a third interface, the third interface including one or more operation options, the one or more operation options including the first operation option; receiving and responding to the user's operation on the first operation option, and determining the operation corresponding to the first gesture as the first operation.

[0674] For example, the third interface can be the operation selection interface 1410 shown in Figure 14B above, in which case the first operation option can be the screen unlock option 1411.

[0675] This way, after creating the first gesture, you can set the operation corresponding to the first gesture as the first operation.

[0676] In one possible implementation, receiving and responding to the user's operation on the first operation option and determining the operation corresponding to the first gesture as the first operation specifically includes: receiving the user's operation on the first operation option; if the security level of the first operation is lower than or equal to the security level of the first gesture, determining the operation corresponding to the first gesture as the first operation.

[0677] In this way, when setting the correspondence between the first gesture and the first operation, it can be determined whether the security level of the first operation is higher than that of the first gesture. If the security level of the first operation is lower than or equal to that of the first gesture, the operation corresponding to the first gesture can be set as the first operation. This can avoid the mismatch between the security of the operation and the gesture and improve the security of the operation triggering conditions.

[0678] In one possible implementation, the first electronic device stores a second gesture; the method further includes: receiving an operation from a user to set a correspondence between the second gesture and a third operation; if the security level of the third operation is higher than the security level of the second gesture, outputting a security prompt, the security prompt being used to indicate to the user that the security level of the third operation is higher than the security level of the second gesture.

[0679] For example, the third operation can be the operation of opening the payment code corresponding to payment code option 1413 in the embodiment shown in FIG15C.

[0680] In this way, the first electronic device can establish a correspondence between the second gesture and the third operation. If the security level of the second gesture is lower than that of the third operation, a security prompt can be output to remind the user that the security level of the third operation is higher than that of the second gesture, so that the user can reselect the gesture corresponding to the third operation or reselect the operation corresponding to the second gesture.

[0681] In one possible implementation, after outputting the security prompt, the method further includes: receiving an operation from the user to change a third operation to a fourth operation; if the security level of the fourth operation is lower than or equal to the security level of the second gesture, determining that the operation corresponding to the second gesture is the fourth operation; or, the electronic device also stores a third gesture; receiving an operation from the user to change a second gesture to a third gesture; if the security level of the third operation is lower than or equal to the security level of the third gesture, determining that the gesture corresponding to the third operation is the third gesture; or, receiving an operation from the user to edit a second gesture, determining the updated second gesture; if the security level of the third operation is lower than or equal to the security level of the updated second gesture, determining that the gesture corresponding to the third operation is the updated second gesture.

[0682] When the security level of the third operation is higher than that of the second gesture, the first electronic device can also refer to the relevant description in the embodiment shown in Figure 9 above to change the operation corresponding to the second gesture, or it can change the gesture corresponding to the third operation, or it can re-edit the second gesture. The methods for determining the security level of the gesture and the security level of the operation can also refer to the relevant description in the embodiment shown in Figure 9, and will not be repeated here.

[0683] Thus, if the security level of the third operation is higher than that of the second gesture, the first electronic device can change the operation corresponding to the second gesture, or it can change the gesture corresponding to the third operation, or it can re-edit the second gesture.

[0684] The following describes a chip system provided by an embodiment of this application.

[0685] This application also provides a chip system including at least one processor for implementing the functions involved in the electronic device 100 in any of the above embodiments.

[0686] In one possible design, the chip system also includes a memory for storing program instructions and data, which may be located within or outside the processor.

[0687] The chip system can consist of chips or include chips and other discrete components.

[0688] Optionally, the chip system may contain one or more processors. These processors can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.

[0689] Optionally, the chip system may contain one or more memories. The memory may be integrated with the processor or disposed separately from it; this application embodiment does not limit this. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or disposed separately on different chips. This application embodiment does not specifically limit the type of memory or the arrangement of the memory and processor.

[0690] For example, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a micro controller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0691] It is understood that the above chip system is only an example. In the embodiments of this application, the chip system may also include more, fewer, or different devices than those in the above embodiments. This application does not limit the scope of the invention.

[0692] The various embodiments of this application can be combined arbitrarily to achieve different technical effects.

[0693] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0694] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0695] In summary, the above description is merely an embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made based on the disclosure of this application should be included within the scope of protection of this application.

Claims

1. A gesture control method, applied to a first electronic device, characterized in that, The first electronic device has multiple pre-stored actions; the method includes: The user's first action was detected; The user's second action was detected; In response to the user ending the first gesture, The first operation corresponding to the first gesture is executed. The first gesture includes the first action and the second action, and the starting action of the first gesture is the first action.

2. The method according to claim 1, characterized in that, The ending action of the first gesture is the second action; The response to the user ending the first gesture specifically includes: No user action was detected during the first time period. The start time of the first time period is the time when the second action was detected, and the duration of the first time period is the first duration.

3. The method according to claim 1, characterized in that, The response to the user ending the first gesture specifically includes: It is determined that among the one or more gestures stored in the first electronic device, there is no gesture that includes the first action and the second action, whose starting action is the first action, and whose number of actions is greater than the number of actions detected.

4. The method according to any one of claims 1-3, characterized in that, After detecting that the user has performed the first action, the method further includes: Output a first prompt, which includes one or more action guides, each of which includes one or more actions, which are actions that the user can perform after the first action.

5. The method according to claim 4, characterized in that, The one or more action guides include a first action guide, and the first action guide includes the second action.

6. The method according to claim 5, characterized in that, The first action guide also includes the name of the first operation, which is used to indicate the operation triggered by the user after performing the first gesture.

7. The method according to claim 5 or 6, characterized in that, The first prompt also includes a first action string, which is used to indicate an action that the user has performed, and the first action string includes the first action.

8. The method according to claim 7, characterized in that, The first prompt output specifically includes: The first action string is displayed based on the first display method; The one or more action guides are displayed based on a second display method, wherein the first display method is different from the second display method.

9. The method according to any one of claims 4-8, characterized in that, After detecting the user's second action, the method further includes: A second prompt is output, which prompts the user that ending the gesture can trigger the first electronic device to perform the first operation.

10. The method according to any one of claims 4-8, characterized in that, The first action guide also includes a third action, and the third action is displayed after the second action; After detecting the user's second action, the method further includes: outputting a third prompt; The third prompt includes a second action guide, which includes the third action; and / or, the third prompt further includes a second action string, which indicates an action that the user has performed, and the second action string includes the first action and the second action.

11. The method according to any one of claims 1-10, characterized in that, Prior to detecting the user's first action, the method further includes: Establish a first communication connection with the second electronic device; The execution of the first operation corresponding to the first gesture specifically includes: A first instruction is sent to the second electronic device through the first communication connection, the first instruction being used to instruct the second electronic device to perform a second operation.

12. The method according to claim 11, characterized in that, The method further includes: The first device type of the second electronic device is obtained based on the first communication connection; Sending the first instruction to the second electronic device through the first communication connection specifically includes: Based on the first device type, a first instruction is sent to the second electronic device through the first communication connection.

13. The method according to any one of claims 1-12, characterized in that, Prior to detecting the user's first action, the method further includes: Display a first interface, the first interface including a first control; Receive and respond to the user's operation on the first control, and display a second interface, the second interface including the first action option, the first action option corresponding to the first action; Receive and respond to the user's operation on the first action option, and display one or more action options, wherein the first or more action options include a second action option, and the second action option corresponds to the second action; Receive and respond to the user's operation on the second action option, determine that the first gesture includes the first action and the second action, and that the first action is the starting action of the first gesture.

14. The method according to claim 13, characterized in that, The second interface also includes a second action option and a third action option, the third action option corresponds to a fourth action, and there is no continuity between the fourth action and the first action; The step of receiving and responding to the user's operation on the first action option, and displaying one or more action options, specifically includes: Upon receiving and responding to the user's action on the first action option, the second action option is displayed, while the third action option is not displayed.

15. The method according to any one of claims 1-14, characterized in that, The first electronic device stores a second gesture; the method further includes: Receives the user's operation to set a correspondence between the second gesture and the third operation; If the security level of the third operation is higher than the security level of the second gesture, a security prompt is output. The security prompt is used to remind the user that the security level of the third operation is higher than the security level of the second gesture.

16. The method according to claim 15, characterized in that, After outputting the security prompt, the method further includes: The user is received requesting that the third operation be changed to the fourth operation. If the security level of the fourth operation is lower than or equal to the security level of the second gesture, the operation corresponding to the second gesture is determined to be the fourth operation. or, The electronic device also stores a third gesture; The user is received to change the second gesture to the third gesture; If the security level of the third operation is lower than or equal to the security level of the third gesture, the gesture corresponding to the third operation is determined to be the third gesture. or, Upon receiving a user's instruction to edit the second gesture, the updated second gesture is confirmed; If the security level of the third operation is lower than or equal to the security level of the updated second gesture, the gesture corresponding to the third operation is determined to be the updated second gesture.

17. An electronic device, specifically a first electronic device, characterized in that: It includes one or more processors and one or more memories; wherein the one or more memories are coupled to one or more processors, and the one or more memories are used to store computer instructions, which, when the one or more processors execute the computer instructions, implement the gesture control method of any one of claims 1-16.

18. A chip system, characterized in that, include: The processing circuit and the interface circuit are provided, wherein the interface circuit is used to receive code instructions and transmit them to the processing circuit, and the processing circuit is used to execute the code instructions to perform the gesture control method according to any one of claims 1-16.

19. A readable storage medium, characterized in that, The device stores computer instructions that, when executed by a processor, implement the gesture control method according to any one of claims 1-16.

20. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the gesture control method according to any one of claims 1-16.

Images