Symbol input method and apparatus based on tapping gesture recognition
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- TSINGHUA UNIVERSITY
- Filing Date
- 2025-12-17
- Publication Date
- 2026-07-30
AI Technical Summary
In the era of spatial computing, existing symbol input schemes have not yet been able to provide a unified and efficient input method, especially in complex real-world scenarios where they are unable to meet users' symbol input needs.
Raw tapping gesture signals are collected by sensors, and gesture recognition is performed using a neural network architecture. By combining finger combinations and postures, symbol input without visual feedback is achieved. Multimodal sensors and preprocessing techniques are used to improve recognition accuracy, and fine-grained gesture sets are introduced to enrich symbol mapping.
It enables natural and intuitive symbol input on any plane without visual feedback, improving input efficiency and accuracy, providing a rich gesture experience, and is suitable for various terminal devices and scenarios.
Smart Images
Figure CN2025143330_30072026_PF_FP_ABST
Abstract
Description
A symbol input method and apparatus based on tapping gesture recognition Technical Field
[0001] This application relates to the field of human-computer interaction technology, and in particular to a symbol input method and apparatus based on tapping gesture recognition. Background Technology
[0002] With the popularity of head-mounted devices such as MR and AR, spatial computing is becoming a reality, leading to deeper levels of human-computer interaction and modern computing methods. Symbol input technology, as a primary input method for interacting with computing devices, is a fundamental requirement when interacting with the spatial environment. In the PC era, the physical keyboard was always the preferred method for symbol input. Its logical layout allowed users to quickly and deftly press their hands and fingers on the operating area. However, in the mobile computing era, while common physical keyboards offer a wealth of keys, they are inconvenient to carry and difficult to use anytime, anywhere. The popularity of smart mobile devices (such as smartphones, tablets, and watches) has made new touchscreen keyboards a more convenient option for everyday text and symbol input. As we move into the era of spatial computing, moving further away from PCs and mobile devices, the need for symbol input will emerge in various environments.
[0003] However, there is currently no unified and efficient input scheme for the complex real-world scenarios that spatial computing may face. Summary of the Invention
[0004] This application provides a symbol input method and apparatus based on tapping gesture recognition, which can be applied to human-computer interaction, such as mobile devices, desktop computers, smart wearable devices, and in-vehicle interaction. It provides a unified and efficient input scheme that does not rely on visual feedback, improves the efficiency and accuracy of symbol input, and provides users with a rich gesture experience.
[0005] In a first aspect, embodiments of this application provide a symbol input method based on tapping gesture recognition, the method comprising:
[0006] The original tapping gesture signal is acquired by a sensor and preprocessed to obtain the tapping gesture signal.
[0007] Gesture recognition is performed based on tapping gesture signals to determine the tapping gesture, which includes the combination of tapping fingers on the tapping plane and the tapping posture of each tapping finger.
[0008] The system uses a preset symbol mapping table to map tapping gestures to symbols and displays the corresponding visual information on the display device.
[0009] This technical solution utilizes a neural network architecture to recognize typing gestures on any plane, including finger combinations and finger postures. This enables typing input and command control on any plane without visual feedback. It operates naturally and intuitively, allowing for typing input on any plane, and introduces a more granular set of gestures, enabling richer symbol mapping schemes to enhance the user's symbol input experience. Compared to ordinary methods of recognizing finger combinations, it innovatively introduces finger postures to assist in improving the representation of finger combination space, thereby expanding the definition space of typing gestures exponentially.
[0010] In one implementation, the sensor includes an image sensor, a depth sensor, and a motion sensor.
[0011] In one implementation, the original tapping gesture signal is preprocessed to obtain a tapping gesture signal, including:
[0012] Filtering techniques are used to remove random noise from the original tapping gesture signal;
[0013] Data standardization is used to scale down the original tapping gesture signals to the same size.
[0014] A time series analysis algorithm is used to perform time-series segmentation on the original tapping gesture signal to determine the tapping gesture signal at the moment of the tapping action.
[0015] In one implementation, gesture recognition is performed based on the tapping gesture signal to determine the tapping gesture, including:
[0016] The tapping gesture signal is input into a neural network architecture based on multimodal sensor data, and the output is the tapping gesture. The construction of the neural network architecture includes:
[0017] The neural network architecture uses an encoder-decoder structure;
[0018] In terms of data encoding, a multi-label method is used to represent the tapping fingers. If any finger is in contact with the plane, the corresponding label is set to 1; otherwise, it is set to 0.
[0019] Training neural network architectures, including:
[0020] A dataset is collected, and the neural network architecture is trained using this dataset. The dataset includes negative class samples. The training process includes:
[0021] During training, a decoding branch is added to the neural network architecture so that the neural network can generate finger poses through the finger pose prediction branch, generate tapping finger combinations through the tapping finger prediction branch, and generate heat map representations of the contact areas through the added decoding branch. The finger contact points are then trained with explicit supervision using the heat map representations of the contact areas.
[0022] During training, the finger pose loss for negative class samples is masked in the optimization function, while the finger tapping combination loss is trained normally.
[0023] In one implementation, the symbols in a pre-defined symbol mapping table correspond to the inputs of the input device, and the inputs of the input device include letters, numbers, punctuation marks, and function commands.
[0024] Secondly, embodiments of this application provide a symbol input device based on tapping gesture recognition, the device comprising:
[0025] The gesture signal acquisition module is used to acquire raw tapping gesture signals through sensors and preprocess the raw tapping gesture signals to obtain tapping gesture signals.
[0026] The gesture recognition module is used to recognize gestures based on tapping gesture signals and determine the tapping gestures, wherein the tapping gestures include the combination of tapping fingers on the tapping plane and the tapping posture of each tapping finger.
[0027] The display module is used to map tapping gestures to symbols using a preset symbol mapping table and display the corresponding visual information on the display device.
[0028] This technical solution utilizes a neural network architecture to recognize typing gestures on any plane, including finger combinations and finger postures. This enables typing input and command control on any plane without visual feedback. It operates naturally and intuitively, allowing for typing input on any plane, and introduces a more granular set of gestures, enabling richer symbol mapping schemes to enhance the user's symbol input experience. Compared to ordinary methods of recognizing finger combinations, it innovatively introduces finger postures to assist in improving the representation of finger combination space, thereby expanding the definition space of typing gestures exponentially.
[0029] In one implementation, the sensor includes an image sensor, a depth sensor, and a motion sensor.
[0030] In one implementation, the original tapping gesture signal is preprocessed to obtain a tapping gesture signal, including:
[0031] Filtering techniques are used to remove random noise from the original tapping gesture signal;
[0032] Data standardization is used to scale down the original tapping gesture signals to the same size.
[0033] A time series analysis algorithm is used to perform time-series segmentation on the original tapping gesture signal to determine the tapping gesture signal at the moment of the tapping action.
[0034] In one implementation, the gesture recognition module is specifically used for:
[0035] The tapping gesture signal is input into a neural network architecture based on multimodal sensor data, and the output is the tapping gesture. The construction of the neural network architecture includes:
[0036] The neural network architecture uses an encoder-decoder structure;
[0037] In terms of data encoding, a multi-label method is used to represent the tapping fingers. If any finger is in contact with the plane, the corresponding label is set to 1; otherwise, it is set to 0.
[0038] Training neural network architectures, including:
[0039] A dataset is collected, and the neural network architecture is trained using this dataset. The dataset includes negative class samples. The training process includes:
[0040] During training, a decoding branch is added to the neural network architecture so that the neural network can generate finger poses through the finger pose prediction branch, generate tapping finger combinations through the tapping finger prediction branch, and generate heat map representations of the contact areas through the added decoding branch. The finger contact points are then trained with explicit supervision using the heat map representations of the contact areas.
[0041] During training, the finger pose loss for negative class samples is masked in the optimization function, while the finger tapping combination loss is trained normally.
[0042] In one implementation, the symbols in a pre-defined symbol mapping table correspond to the inputs of the input device, and the inputs of the input device include letters, numbers, punctuation marks, and function commands. Attached Figure Description
[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.
[0044] Figure 1 is a flowchart illustrating a symbol input method based on tapping gesture recognition provided in an embodiment of this application;
[0045] Figure 2 is an example diagram of a predefined set of tapping gestures provided in an embodiment of this application;
[0046] Figure 3 is a schematic diagram of the training process of the neural network architecture based on multimodal sensor data provided in the embodiments of this application;
[0047] Figure 4 is an example diagram of the tapping gesture symbol mapping provided in an embodiment of this application;
[0048] Figure 5 is a schematic diagram of the structure of a communication device provided in an embodiment of this application. Detailed Implementation
[0049] The symbol input method and apparatus based on tapping gesture recognition provided in this application will be described in detail below with reference to the accompanying drawings.
[0050] Please refer to Figure 1, which is a flowchart illustrating a symbol input method based on tapping gesture recognition provided in an embodiment of this application. As shown in Figure 1, the method may include, but is not limited to, the following steps:
[0051] Step 101: Collect the original tapping gesture signal through the sensor, and preprocess the original tapping gesture signal to obtain the tapping gesture signal;
[0052] Step 102: Perform gesture recognition based on the tapping gesture signal to determine the tapping gesture, wherein the tapping gesture includes the combination of tapping fingers on the tapping plane and the tapping posture of each tapping finger;
[0053] Step 103: Use a preset symbol mapping table to map the tapping gestures to symbols, and display the corresponding visual information on the display device;
[0054] Optionally, by integrating various types of sensors, such as image sensors, depth sensors, and motion sensors, into different locations on the wristband, signals from different tapping gestures of the user can be collected (which can be a single complete hand image, multiple continuous motion images, or time-series data from the motion sensor throughout the entire movement). The collected image signals can include images from above or below the hand, depending on the wearing position of the image acquisition sensor (including but not limited to the wrist, arm, and head), requiring the image sensor to completely capture the entire tapping finger when the user performs the tapping gesture.
[0055] Optionally, the preprocessing steps include:
[0056] Filtering techniques are used to remove random noise from signals such as images and motion sensors.
[0057] Because the signals from motion sensors, image sensors, and other sensors are unevenly distributed across multiple dimensions, data standardization is used to scale the signals to the same scale.
[0058] A time-series analysis algorithm is used to perform temporal segmentation on the gesture signal, and the image signal at the actual tapping moment is used for subsequent fine-grained tapping gesture recognition steps. Specifically, this process includes:
[0059] When a user taps a flat surface with their finger, a time-series analysis algorithm acquires a processed tapping gesture image. When multiple consecutive tapping images exist, the sequence is further subdivided to identify image frames in the gesture sequence where the finger image is clear and the movement is complete and obvious.
[0060] Optionally, the gesture recognition process includes:
[0061] Key features of finger contact with a flat surface in the spatial or frequency domain of the processed tapping gesture signal can be extracted (e.g., fingertip position, flat surface location, degree of finger contour deformation, image intensity changes, image frequency response, etc.). Furthermore, deep feature extraction operators (including but not limited to convolutional networks, residual networks, sampling modules, etc.) can be used to extract deep features from the original or feature signals, thereby revealing subtle, deep gesture features that are difficult to detect directly.
[0062] A pattern recognition algorithm is applied to classify the input signal in order to identify and distinguish various gestures in a predefined set of fine-grained gestures, which are mainly composed of multiple finger combinations and different finger postures. The predefined set of tapping gestures is shown in Figure 2, which is defined by three finger postures and all finger combinations.
[0063] Accurate calibration of contact positions is crucial when studying discrete finger tapping combinations and postures. Contact positions contain rich regional information and are core features for gesture classification and discrimination. Therefore, to enhance the model's focus on contact areas, a dedicated supervision branch is designed in the pattern recognition algorithm to extract and strengthen contact area information, ensuring that the neural network focuses on key contact area features rather than being interfered with by image textures or other irrelevant environmental factors. Specifically, a neural network architecture based on multimodal sensor data is proposed. This architecture learns feature representations of the input data through an encoder-decoder module to predict finger combinations and postures. Simultaneously, to explicitly supervise contact area feature extraction, a dedicated decoding branch is added to the network. This branch generates heatmap representations of the contact areas for explicit supervision of finger contact points. This significantly increases the weight of contact areas in the feature extraction process, thereby optimizing the neural network's representational ability. This design, by combining multimodal data fusion with explicit contact area supervision, effectively improves the network's feature capture ability for tapping gestures, reduces interference from environmental factors, and enhances the model's robustness and generalization ability. The specific training process is shown in Figure 3.
[0064] Optionally, negative class samples in the dataset also need to be collected to cover various complex but non-tapping gestures in the real world. This design aims to improve the model's robustness and discrimination ability in diverse environments, thereby effectively distinguishing non-tapping gestures that may occur in real-world scenarios. For data encoding, a multi-label approach is used to represent finger states. Specifically, each finger is marked as 1 if it is in contact with a plane, and 0 otherwise. This encoding method can accurately describe the contact state of the fingers and provide rich feature information for the model. On the other hand, since negative class samples do not involve tapping actions, their finger pose information cannot be clearly defined. During model training, to avoid invalid supervision interfering with the optimization process, the finger pose loss of negative class samples is masked in the optimization function, and only the neural network branch for finger combination prediction is trained normally. This strategy ensures that the model can focus on finger combination prediction while reducing the impact of pose classification uncertainty caused by negative class samples on the overall training.
[0065] Based on pattern recognition results, the system determines the combination of finger tapping in the current pressing gesture (e.g., pressing down with the thumb, middle finger, and ring finger simultaneously) and the finger tapping posture (e.g., tapping the table with the fingertip). This is quantified into three intervals: the angle between the tapping finger and the table is less than 90 degrees, approximately 90 degrees, and greater than 90 degrees. Of course, the fingertip tapping posture is just an example; the specific tapping location needs to be adjusted according to user needs and habits. Specifically, users can define and divide the entire posture space according to their own tapping needs, selecting different areas of the fingers for tapping. This includes not only the front of the hand but also various possible tapping contact areas such as the nails and the sides of the fingers. Furthermore, multiple sensors (including but not limited to both wrists, arms, and head) can be worn in multiple different locations, or external image sensing devices (including but not limited to mobile phones and computers) can be used to support the recognition of fine-grained tapping gestures from multiple fingers of both hands.
[0066] Optionally, the symbol mapping process includes:
[0067] Based on the identified fine-grained tapping gestures, a preset symbol mapping table is used to map them to symbols. During symbol mapping, fine-grained tapping gestures, composed of finger tapping combinations and gestures, can be used as input to map various possible symbols. It should be further noted that the symbol mapping method is related to the size of the contact area. If the contact plane is small, supporting only a few fingers instead of all fingers (e.g., the contact plane can only accommodate three fingers), then only consecutive finger combinations of fewer than three fingers are considered for key mapping. If the contact plane is narrow, supporting only specific finger gestures, then other finger gestures can be disregarded, degenerating into key mapping based solely on finger combinations. If the finger contact area is larger, allowing more fingers and gestures to tap simultaneously, a finer mapping division can be performed, resulting in more selectable fine-grained tapping gestures. From these, a symbol mapping method that better suits the user's specific needs and is ergonomic can be chosen.
[0068] There are countless possibilities for symbol mapping. Below is an example of a possible text input mapping method for a relatively large contact area. The mapping table is shown in Figure 4. Recognizable fine-grained typing gestures can arbitrarily define letters. Note that the mapping layout is not fixed; users can map letters according to their own comfortable and habitual gestures. First, the comfort of typing a single letter was considered, so letter mapping was performed for five single fingers and three finger postures. Then, two-finger and three-finger gestures were used to map the remaining letters in turn. In this embodiment, letters are sorted from high to low frequency (i.e., the average number of times a letter appears in written language), and single-finger, two-finger, and three-finger gestures are used for mapping in turn.
[0069] Optionally, in order to achieve input error correction and association functions, statistical models of natural language can also be used to correct and associate the input symbol combinations.
[0070] Optionally, Figure 4 only lists some of the symbols. Other symbols and command control mappings can be defined using fine tapping gestures composed of all other finger combinations and three tapping postures.
[0071] Optionally, the display process includes:
[0072] The system outputs real-time visual feedback information of symbol input through various means such as visualization (displaying symbol mapping results), function activation prompts, vibration, etc., on the graphical interface of various device screens (including but not limited to VR / AR headsets, computers, mobile phones, tablets, and watches). The system can present the user's input in the form of text, symbols, images, etc., and dynamically update the displayed content according to the real-time changes in user interaction.
[0073] Optionally, the display process may also include:
[0074] The system locates the specific position of the input symbol during the symbol input process, thereby comprehensively improving the accuracy and convenience of input in various symbol input application scenarios.
[0075] By implementing the embodiments of this application, gesture signals are collected using accelerometers, gyroscopes, magnetometers, light sensors, touch sensors, image sensors, and depth sensors. The combination of these sensors can effectively capture multi-dimensional gesture data, providing more accurate gesture recognition capabilities. Furthermore, by integrating data from different sensors, the robustness and adaptability of the system can be improved to cope with diverse usage scenarios. Preprocessing of the signals is performed using algorithms such as inter-frame differencing, data denoising, normalization, illumination contrast adjustment, edge detection, and data fusion. For fine tapping gesture recognition, feature extraction is performed using algorithms such as deep learning descriptors, scale-invariant feature transform (SIFT), time series analysis, optical flow, and short-time Fourier transform (STFT) based on frequency domain analysis. This allows for the extraction of key features from complex gesture data, capturing subtle changes in gestures, and enhancing the accuracy and robustness of recognition. Finally, a custom symbol mapping relationship is used to implement tapping gesture symbol mapping, effectively enhancing the user experience.
[0076] By implementing the embodiments of this application, the hardware carrier becomes applicable to various terminal electronic devices, including but not limited to mobile phones, smartwatches, tablets, personal computers, smart glasses, VR headsets, and other devices. Furthermore, the hardware carrier can also be applied to IoT devices, smart home systems, in-vehicle electronic devices, and medical monitoring instruments. By combining with different types of terminal devices, this invention provides broad compatibility and flexibility, meeting users' symbol input needs in various scenarios. Whether in daily life, sports and fitness, entertainment and interaction, or professional applications, the hardware carrier can fully leverage its performance advantages, bringing users a more convenient and efficient experience.
[0077] Please refer to Figure 5, which is a schematic diagram of a symbol input device based on tapping gesture recognition provided in an embodiment of this application. The communication device shown in Figure 5 includes:
[0078] The gesture signal acquisition module is used to acquire raw tapping gesture signals through sensors and preprocess the raw tapping gesture signals to obtain tapping gesture signals.
[0079] The gesture recognition module is used to recognize gestures based on tapping gesture signals and determine the tapping gestures, wherein the tapping gestures include the combination of tapping fingers on the tapping plane and the tapping posture of each tapping finger.
[0080] The display module is used to map tapping gestures to symbols using a preset symbol mapping table and display the corresponding visual information on the display device.
[0081] Optionally, the sensors include image sensors, depth sensors, and motion sensors.
[0082] Optionally, the original tapping gesture signal is preprocessed to obtain a tapping gesture signal, including:
[0083] Filtering techniques are used to remove random noise from the original tapping gesture signal;
[0084] Data standardization is used to scale down the original tapping gesture signals to the same size.
[0085] A time series analysis algorithm is used to perform time-series segmentation on the original tapping gesture signal to determine the tapping gesture signal at the moment of the tapping action.
[0086] Optional, gesture recognition module, specifically used for:
[0087] The tapping gesture signal is input into a neural network architecture based on multimodal sensor data, and the output is the tapping gesture. The construction of the neural network architecture includes:
[0088] The neural network architecture uses an encoder-decoder structure;
[0089] In terms of data encoding, a multi-label method is used to represent the tapping fingers. If any finger is in contact with the plane, the corresponding label is set to 1; otherwise, it is set to 0.
[0090] Training neural network architectures, including:
[0091] A dataset is collected, and the neural network architecture is trained using this dataset. The dataset includes negative class samples. The training process includes:
[0092] During training, a decoding branch is added to the neural network architecture so that the neural network can generate finger poses through the finger pose prediction branch, generate tapping finger combinations through the tapping finger prediction branch, and generate heat map representations of the contact areas through the added decoding branch. The finger contact points are then trained with explicit supervision using the heat map representations of the contact areas.
[0093] During training, the finger pose loss for negative class samples is masked in the optimization function, while the finger tapping combination loss is trained normally.
[0094] Optionally, the symbols in the preset symbol mapping table correspond to the inputs of the input device, which include letters, numbers, punctuation marks, and function commands.
[0095] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer programs. When the computer program is loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program can be transferred from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).
[0096] Those skilled in the art will understand that the various numerical designations such as "first," "second," etc., involved in this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application, nor do they indicate the order of sequence.
[0097] At least one in this application can also be described as one or more, and multiple can be two, three, four or more, and this application does not impose any limitation. In the embodiments of this application, for a technical feature, the technical features in that technical feature are distinguished by "first", "second", "third", "A", "B", "C" and "D", and there is no order or size among the technical features described by "first", "second", "third", "A", "B", "C" and "D".
[0098] The correspondences shown in the tables of this application can be configured or predefined. The values of the information in each table are merely examples and can be configured to other values; this application is not limited to these values. When configuring the correspondences between information and parameters, it is not necessarily required to configure all the correspondences shown in each table. For example, the correspondences shown in some rows of the tables in this application may not be configured. Furthermore, appropriate modifications and adjustments can be made based on the above tables, such as splitting, merging, etc. The names of the parameters shown in the headings of the above tables can also use other names that the communication device can understand, and the values or representations of the parameters can also be other values or representations that the communication device can understand. In the implementation of the above tables, other data structures can also be used, such as arrays, queues, containers, stacks, linear lists, pointers, linked lists, trees, graphs, structures, classes, heaps, hash tables, or hash tables, etc.
[0099] The term "predefined" in this application can be understood as definition, pre-defined, stored, pre-stored, pre-negotiated, pre-configured, solidified, or pre-burned.
[0100] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0101] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0102] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A symbol input method based on tapping gesture recognition, characterized in that, The method includes: The original tapping gesture signal is acquired by a sensor and preprocessed to obtain the tapping gesture signal. Gesture recognition is performed based on the tapping gesture signal to determine the tapping gesture, wherein the tapping gesture includes the combination of tapping fingers on the tapping plane and the tapping posture of each tapping finger; The tapping gestures are mapped to symbols using a preset symbol mapping table, and the corresponding visual information is displayed on a display device.
2. The method according to claim 1, characterized in that, The sensors include image sensors, depth sensors, and motion sensors.
3. The method according to claim 1, characterized in that, The preprocessing of the original tapping gesture signal to obtain the tapping gesture signal includes: Filtering techniques are used to remove random noise from the original tapping gesture signal; Data standardization is used to scale down the original tapping gesture signals to the same size. A time series analysis algorithm is used to perform time-series segmentation on the original tapping gesture signal to determine the tapping gesture signal at the moment of the tapping action.
4. The method according to claim 1, characterized in that, Based on the tapping gesture signal, gesture recognition is performed to determine the tapping gesture, including: The tapping gesture signal is input into a neural network architecture based on multimodal sensor data, and the tapping gesture is output. The construction of the neural network architecture includes: The neural network architecture uses an encoder-decoder structure; In terms of data encoding, a multi-label method is used to represent the tapping fingers. If any finger is in contact with the plane, the corresponding label is set to 1; otherwise, it is set to 0. Training the neural network architecture includes: A dataset is collected, and the neural network architecture is trained using the dataset. The dataset includes negative class samples. The training process includes: During training, a decoding branch is added to the neural network architecture so that the neural network can generate finger poses through the finger pose prediction branch, generate tapping finger combinations through the tapping finger prediction branch, and generate heat map representations of the contact areas through the added decoding branch. The finger contact points are then trained with explicit supervision using the heat map representations of the contact areas. During training, the finger pose loss for negative class samples is masked in the optimization function, while the finger tapping combination loss is trained normally.
5. The method according to claim 1, characterized in that, The symbols in the preset symbol mapping table correspond to the inputs of the input device, and the inputs of the input device include letters, numbers, punctuation marks, and function commands.
6. A symbol input device based on tapping gesture recognition, characterized in that, The device includes: The gesture signal acquisition module is used to acquire raw tapping gesture signals through sensors and preprocess the raw tapping gesture signals to obtain tapping gesture signals. A gesture recognition module is used to perform gesture recognition based on the tapping gesture signal and determine the tapping gesture, wherein the tapping gesture includes a combination of tapping fingers on the tapping plane and the tapping posture of each tapping finger; The display module is used to map the tapping gestures to symbols using a preset symbol mapping table, and to display the corresponding visual information on the display device.
7. The apparatus according to claim 6, characterized in that, The sensors include image sensors, depth sensors, and motion sensors.
8. The apparatus according to claim 6, characterized in that, The preprocessing of the original tapping gesture signal to obtain the tapping gesture signal includes: Filtering techniques are used to remove random noise from the original tapping gesture signal; Data standardization is used to scale down the original tapping gesture signals to the same size. A time series analysis algorithm is used to perform time-series segmentation on the original tapping gesture signal to determine the tapping gesture signal at the moment of the tapping action.
9. The apparatus according to claim 6, characterized in that, The gesture recognition module is specifically used for: The tapping gesture signal is input into a neural network architecture based on multimodal sensor data, and the tapping gesture is output. The construction of the neural network architecture includes: The neural network architecture uses an encoder-decoder structure; In terms of data encoding, a multi-label method is used to represent the tapping fingers. If any finger is in contact with the plane, the corresponding label is set to 1; otherwise, it is set to 0. Training the neural network architecture includes: A dataset is collected, and the neural network architecture is trained using the dataset. The dataset includes negative class samples. The training process includes: During training, a decoding branch is added to the neural network architecture so that the neural network can generate finger poses through the finger pose prediction branch, generate tapping finger combinations through the tapping finger prediction branch, and generate heat map representations of the contact areas through the added decoding branch. The finger contact points are then trained with explicit supervision using the heat map representations of the contact areas. During training, the finger pose loss for negative class samples is masked in the optimization function, while the finger tapping combination loss is trained normally.
10. The apparatus according to claim 6, characterized in that, The symbols in the preset symbol mapping table correspond to the inputs of the input device, and the inputs of the input device include letters, numbers, punctuation marks, and function commands.