Rapid gesture recognition method based on fusion of spatiotemporal features of transient electromyographic signals and force sensor signals

By fusing the spatiotemporal characteristics of electromyographic signals with force sensor signals, employing three-dimensional feature extraction and selective attention mechanisms, and combining force signal regression and physiological dynamic constraints, the problems of low accuracy and weak generalization ability in electromyographic signal gesture recognition are solved, achieving high accuracy and fast response electromyographic prosthetic control.

WO2026233472A1PCT designated stage Publication Date: 2026-11-12SOUTHEAST UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2026/105165
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-09-29
Filing Date
2026-06-23
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Existing gesture recognition technologies based on electromyography signals have low accuracy in multi-gesture classification tasks, and the models lack objective signal-assisted supervised training and physiological constraints, resulting in weak generalization ability and failing to meet the practical application requirements of fast response and high accuracy.

Method used

By fusing the spatiotemporal features of transient electromyography signals with force sensor signals, three-dimensional features are extracted using spatial gradient and multi-scale temporal difference. A phase-physiological joint temporal coding and frame-by-frame force signal-guided selective attention mechanism are introduced, and the model is trained by combining force signal regression and physiological dynamics constraints.

Benefits of technology

It improves the accuracy of multi-gesture classification tasks, enhances the model's generalization ability and interpretability, and can better adapt to the myoelectric prosthesis control needs of different users.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2026105165_12112026_PF_FP_ABST
    Figure CN2026105165_12112026_PF_FP_ABST
Patent Text Reader

Abstract

A rapid gesture recognition method based on fusion of spatiotemporal features of transient electromyographic signals and force sensor signals. Spatiotemporal features are extracted and force sensor signals are incorporated for auxiliary supervision during training. The method comprises a training phase and a testing phase. In the training phase, preprocessing operations, such as electrode interference detection, damaged-electrode repair and normalization, are first performed on transient high-density electromyographic signals; three-dimensional spatiotemporal features of the electromyographic signals are then constructed, and phase-physiological joint temporal encoding is also performed; the three-dimensional spatiotemporal features and a phase-physiological joint temporal encoding are combined, and the combination result is fed into a joint model based on a selective attention mechanism; and a weighted total loss comprising a cross-entropy classification loss, a force signal regression loss and a physiological dynamics loss is minimized to perform optimization. In the testing phase, only transient electromyographic signals are input; and after preprocessing and the extraction of spatiotemporal features, a trained model is used to complete gesture classification.
Need to check novelty before this filing date? Find Prior Art

Description

A method for rapid gesture recognition by fusing the spatiotemporal features of transient electromyography signals with force sensor signals Technical Field

[0001] This invention relates to the interdisciplinary field of biosignal processing and pattern recognition, specifically to the application of electromyography (EMG) signal analysis and gesture recognition. It can be directly used for the precise control of EMG prostheses, the control of motion assistive devices in rehabilitation medicine, and the signal processing module of human-computer interaction systems (such as wearable gesture control terminals), providing core technical support for scenarios that require the conversion of gesture commands through EMG signals. Background Technology

[0002] Currently, gesture recognition technology based on electromyography (EMG) signals has been widely applied in fields such as prosthetic limb control and human-computer interaction. Among them, transient EMG signals (i.e., short-term signals during the initiation phase of gesture movements) have become a key research object for achieving "rapid recognition" because they can reflect the early characteristics of gestures. However, existing technologies still have two major shortcomings:

[0003] 1. Insufficient spatiotemporal feature extraction capability leads to low accuracy in large gesture classification tasks: Current methods that utilize transient electromyographic signals during gesture execution are limited to single temporal or spatial features, ignoring spatial gradient features and multi-timescale features. As a result, when the number of gestures is relatively small (less than 20), the accuracy is acceptable, but when the number of gestures is large (more than 60), the accuracy is low.

[0004] 2. The lack of objective signal-assisted supervised training and physiological constraints leads to low model interpretability and weak generalization ability: Existing electromyography data-driven models only focus on classification results. The training process of model parameters lacks objective signal-assisted supervised training and physiological constraints, resulting in poor model interpretability and weak generalization ability.

[0005] The aforementioned deficiencies prevent existing technologies from simultaneously meeting the practical application requirements of "rapid response, high accuracy, and strong generalization." In particular, in the control of myoelectric prostheses, problems such as movement delay and command misjudgment are prone to occur, affecting the user experience and rehabilitation effect. Therefore, there is an urgent need for a gesture recognition solution that can take into account the spatiotemporal characteristics of transient myoelectric signals, has strong anti-interference ability, and excellent generalization. Summary of the Invention

[0006] This invention provides a method and system for rapid gesture recognition that integrates the spatiotemporal features of transient electromyographic signals with force sensor signals. Through a full-process design of "data preprocessing - feature extraction - joint model construction and training - test recognition", it innovatively proposes a spatiotemporal three-dimensional feature based on spatial gradient and multi-scale temporal difference, as well as a force signal-assisted supervised training method based on physiological constraints.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows: a method for rapid gesture recognition by fusing the spatiotemporal characteristics of transient electromyographic signals with force sensor signals, the method comprising the following steps:

[0008] S1 Training Data Acquisition and Preprocessing

[0009] S1.1 Data Acquisition and Time Stamping

[0010] Electromyography (EMG) signal acquisition, force signal acquisition,

[0011] Time synchronization: While acquiring electromyographic and force signals, a synchronization trigger unit marks the two signals for time synchronization, and the processing unit then synchronizes the two signals based on this mark.

[0012] S1.2 Bandpass and Notch Filtering

[0013] Bandpass filtering: A 30Hz-500Hz bandpass filter is used.

[0014] Notch filtering: A notch filter with a center frequency of 50Hz is used.

[0015] S1.3 Electrode Interference Detection and Damaged Electrode Repair

[0016] The electrode interference detection method is as follows: when the amplitude of a certain signal segment exceeds a set threshold ε, it is marked as interference: max(|x i [n]|)>εσ i

[0017] Where, x i [n] represents the signal amplitude, σ i This represents the standard deviation of the current segment signal.

[0018] The method for repairing damaged electrodes is through neighborhood bilinear interpolation:

[0019] in, For the channel being repaired, x (u,v) [n] represents the nearest passage, w uv To fix the weights.

[0020] S1.4 Standardization: Ensures consistent data distribution.

[0021] Specifically:

[0022] S1.4.1. Calculate the training set statistics, including the mean and variance.

[0023] S1.4.2. Based on the mean and variance, standardize the signal.

[0024] S1.4.3. Save the obtained mean and variance to prepare for the reuse of test data.

[0025] In this step, electrode interference detection and damaged electrode repair are performed using s1.3. This can remove outlier interference and improve classification accuracy during subsequent spatial feature extraction.

[0026] Construction of spatiotemporal features of S2 electromyographic signals

[0027] S2.1 Electromyography array interpolation resampling and boundary removal

[0028] Interpolation resampling: For low-channel EMG arrays, bilinear interpolation is used to expand them to a higher number of channels. Boundary removal: Remove the outermost channels of the array and retain the effective channels in the middle.

[0029] S2.2 Spatiotemporal gradient calculation and spatiotemporal feature construction

[0030] The specific steps are as follows:

[0031] S2.2.1, 3D spatiotemporal gradient calculation, the specific calculation formula is as follows:

[0032] (a) Horizontal gradient G x (i,j,t)=EMG′(i+1,j,t)-EMG′(i-1,j,t)

[0033] (b) Vertical gradient G y (i,j,t)=EMG′(i,j+1,t)-EMG′(i,j-1,t)

[0034] (c) Multiscale temporal (causal) difference

[0035] Among them, different Δt k Used to capture changes in electromyographic signals at different time scales

[0036] (d) Synthesize the three-dimensional gradient vector at each position (i,j,t):

[0037] Where η is a scaling factor balancing the spatial and temporal dimensions, typically taken as 0.5–2, and can be learned automatically through experiments or training.

[0038] (e) Calculate the gradient magnitude:

[0039] (f) Spatial orientation angle θ(i,j,t) (plane angle):

[0040] (g) Time tilt angle φ(i,j,t) (angle of spatiotemporal variation):

[0041] S2.2.2. Construction of three-dimensional spatiotemporal features

[0042] (a) Divide the current frame into n subframes according to certain rules. The following steps (b) and (c) are performed in the subframes. (d) Stitch all the subframes together to obtain the three-dimensional spatiotemporal features.

[0043] The specifications can be: irregularly divided according to muscle regions (flexor region, extensor region, pronation and supination region, etc.), or fixedly divided according to channel position (dividing the 16*16 channel topology into 16 4*4 channel topologies).

[0044] (b) Discretize θ(i,j,t) and φ(i,j,t) in two dimensions: Discretize the spatial orientation angle θ(i,j,t) into K θ The time tilt angle φ(i,j,t) is discretized into K intervals. φ There are several intervals. For each time t, the sum of the amplitudes within each interval is calculated:

[0045] Wherein, H(k) θ ,k φ (t) represents the spatiotemporal orientation distribution within the current time frame, with an output dimension of K. θ ×K φ Example: If K θ =8,K φ =4, then each frame outputs 32-dimensional features.

[0046] (c) Spatiotemporal feature construction: The statistical results of all times are concatenated in chronological order to obtain the final shape [T,K]. θ ×K φ The spatiotemporal feature block of the m-th subframe: f m =[H(·,·,1),H(·,·,2),…,H(·,·,T)]

[0047] (d) Construction of 3D spatiotemporal features: The statistical results of all subframes (m = 1, 2, ..., n) are concatenated according to the splitting order in (a) to obtain the final 3D spatiotemporal features:

[0048] Two-dimensional HOG features can extract spatial gradient information. This step, by adding "multi-scale temporal (causal) difference" and "temporal tilt angle φ(i,j,t)", can obtain temporal variation features and extract three-dimensional electromyographic signal change information.

[0049] S3 Joint Model Construction and Training

[0050] S3.1 Phase-Physiological Joint Temporal Encoding, the specific calculation process is as follows:

[0051] A phase encoding φ t :

[0052] The clip function will Mapped to the interval (0, 1), t onset At the start time, T win This represents the total transient time range.

[0053] B-force perception weight w t

[0054] in, These are frame-by-frame force prediction values, where ∈ represents a small amount required for normalization to improve numerical stability. C yields the final joint temporal coding.

[0055] Step s3.1 Introducing phase encoding φ during the time encoding process t With force perception weight w t Dynamic time position code was generated. Compared to traditional static time coding, by using the force exertion time t... onset Phase encoding φ t The solution addresses the inconsistency in the exact timing of force application by using predicted values. Force perception weight w t This approach addresses the issue of inconsistent force application speed by allowing the model to dynamically adapt to the timing and speed of force application in different samples, focusing more on muscle state information rather than static time information. This design significantly improves the model's generalization ability by accurately capturing changes in muscle state during gesture execution.

[0056] S3.2 Modeling of the selective attention mechanism guided by frame-by-frame force signals, the specific steps are as follows:

[0057] S3.2.1 The spatiotemporal feature block obtained from s2.3 Joint time coding obtained with s3.1 By splicing the data along the time dimension, we obtain the spatiotemporal fusion feature X. t Subsequent operations are based on the spatiotemporal fusion feature X. t Expand.

[0058] S3.2.2 Calculate Q, K, and V values, and calculate the input spatiotemporal fusion feature X. t Through three sets of linear transformations, we obtain: Q = X t W Q K = X t W K V = X t WV

[0059] Among them, W Q W K W V These are the learnable parameters for the query vector Q, key vector K, and value vector V, which are randomly initialized during training.

[0060] S3.2.3 Constructing a time-weighted square matrix M based on predictive power

[0061] Among them, M ij σ is the element in the i-th row and j-th column of the square matrix M, representing the difference between the i-th and j-th frames in the "force signal" dimension, and σ(·) represents the normalization function.

[0062] S3.2.4 Improved attention calculation formula:

[0063] Where d k Let K be the feature dimension of Q in the attention calculation process, and ρ be the constraint strength of the control force signal. S3.2.5 Output feature: Z = AV

[0064] Where A is the attention weight matrix calculated in S3.2.4, and V is the value vector calculated in S3.2.2.

[0065] S3.2.6 Force Signal Regression Branch

[0066] After the feature vector of each frame, a linear regression layer is used to output the corresponding force prediction value: f t =W force Z t +b force

[0067] Among them, W force Let b be the weight matrix. force For bias terms,

[0068] S3.2.7 Gesture Classification Branch

[0069] First, perform intra-frame average pooling, then perform cross-frame fully connected layers, and finally output the gesture prediction value:

[0070] (a) Intra-frame average pooling:

[0071] Among them, Z t Let h represent the features of frame t, h·w represent the feature dimension of each frame, and s represent the features of frame t. t The features are the pooled features for each frame.

[0072] (b) Frame by frame stitched into a vector:

[0073] Where T represents the total time dimension.

[0074] (c) Fully connected mapping and probability output: o = W c s+b c

[0075] Among them, W c Let b be the weight matrix of the fully connected network. c For classification bias, o represents the unnormalized score of each category, C represents the total number of categories, and o c To predict the unnormalized score for class c, This is the probability of predicting the class as c.

[0076] Given the "short-lived, rapidly changing" characteristics of transient electromyographic signals, it is necessary to reduce the computational weight for slowly changing ordinary frames and increase the computational weight for rapidly changing key frames. Therefore, based on the traditional self-attention mechanism, this scheme designs a selective attention mechanism guided by frame-by-frame force signals to reduce interference from ordinary frames and improve the selectivity of key frames.

[0077] S3.3 Force signal-based supervised training: The key to training lies in constructing the loss function. The overall joint loss function is based on the basic classification crossover loss L. class Above this, increase the force regression constraint loss L force and physiological and dynamic constraint loss L physio :

[0078] S3.3.1 Classification Cross-Loss

[0079] Where C represents the total number of categories, and y c It is the one-hot encoding of the real label. It is the probability that the model predicts it to be class c.

[0080] S3.3.2 Force Regression Constraint:

[0081] Where T is the total number of time frames. For true strength, For predictive power,

[0082] S3.3.3 Physiological and dynamic constraints,

[0083] To satisfy the law of force change during muscle contraction, three constraints are designed for force prediction in each frame:

[0084] a) Use "smoothness constraint" to constrain the drastic changes between two adjacent frames:

[0085] b) Use "monotonicity constraint" to constrain the change trend between adjacent frames:

[0086] c) Use "rate limit constraint" to constrain the rate of change between adjacent frames:

[0087] Where T is the total number of time frames. The predicted value of the force at time t-1. R is the predicted force value at time t. max The physiological upper limit of force change between two adjacent frames

[0088] Therefore, the total physiological dynamics constraint loss should simultaneously consider the above three constraint losses, and is defined as: L physio =λ s L smooth +λ m L mono +λ r L rate

[0089] Where, λ s , λ m , λ r These are the weights of the three constraints.

[0090] S3.3.4 The overall joint loss function, based on the classification crossover loss L class Above this, increase the force regression constraint loss L force and physiological and dynamic constraint loss L physio :

[0091] L total =αL class +βL force +γL physio

[0092] Where α, β, and γ are the weights of the three types of loss.

[0093] Traditional approaches focus solely on classification loss, neglecting regression loss for objective auxiliary signals and failing to consider physiological changes during action execution. This approach introduces both force signal regression loss and physiological constraints. By applying smoothness, monotonicity, and rate caps, the force prediction results are constrained. This enhances the model's interpretability through auxiliary signals (force signals) and allows the model to focus on force signals rather than simple gesture classification during training, mitigating overfitting and improving generalization.

[0094] S4.1 Test Data Preprocessing,

[0095] Data Acquisition and Preprocessing: Only transient electromyographic signals from the subjects were collected, following the same acquisition procedure as in S1. The standardization step reused the training set statistics μ saved in S1.4. train With σ train Standardize the electromyographic signals during the testing phase to avoid data distribution deviation.

[0096] S4.2 Spatiotemporal Feature Extraction:

[0097] Reuse the training process to obtain spatiotemporal features

[0098] S4.3 Gesture Classification Based on the Model During Training

[0099] Only electromyographic information is used. Based on the spatiotemporal feature extraction of s2.2 and the phase-physiological joint temporal encoding of s3.1, the spatiotemporal fusion feature X is obtained. t The model uses a selective attention mechanism guided by frame-by-frame force signals from input S3.2 to obtain gesture classification results. Simultaneously, it outputs force information frame-by-frame as a reference, improving the model's interpretability.

[0100] Compared with the prior art, the advantages of the present invention are as follows:

[0101] 1. Extracting 3D spatiotemporal features to further improve the accuracy of multi-gesture classification tasks. For electromyography (EMG) data at each time step, the spatial gradient and temporal (causal) difference are calculated. Then, spatial angles and temporal tilt angles are categorized and statistically analyzed to obtain 3D spatiotemporal features, extracting EMG signal changes in the temporal dimension. This method can deeply utilize transient EMG signal features during gesture execution, further improving the accuracy of multi-gesture classification tasks.

[0102] 2. Joint model construction to improve model generalization. To address the inconsistency in the timing and speed of force application among internal samples of the same gesture, phase encoding φ is introduced. t With force perception weight w t Generate phase-physiological joint time codes Furthermore, based on the traditional self-attention mechanism, a selective attention mechanism guided by frame-by-frame force signals was designed, increasing the computational weights for rapidly changing keyframes. This model can accurately capture muscle state changes during gesture execution, thereby improving the model's generalization ability.

[0103] 3. Force-signal-based supervised training enhances model interpretability and suppresses overfitting. To address the issues of poor interpretability and overfitting in machine learning, objective force signals are introduced as constraints on regression loss. Simultaneously, physiological dynamics constraints during gesture execution are incorporated. This ensures the model focuses not only on gesture classification results during training but also on the underlying physiological dynamics-based force information, thereby improving interpretability and suppressing overfitting. Attached Figure Description

[0104] Figure 1: Flowchart of the gesture recognition method, which shows the overall process of training and testing of the present invention.

[0105] Figure 2: Schematic diagram of the gesture recognition system, which shows the overall structure of the system.

[0106] Figure 3: Flowchart of 3D spatiotemporal feature construction. This diagram details the steps involved in constructing 3D spatiotemporal features.

[0107] Figure 4: Flowchart of joint model construction and training, which shows the internal structure of the joint model. Detailed Implementation

[0108] To enhance understanding of the present invention, the embodiments will be described in detail below with reference to the accompanying drawings.

[0109] Example: A method for rapid gesture recognition by fusing the spatiotemporal features of transient electromyography signals with force sensor signals. The entire process is shown in Figure 1. The method includes the following steps:

[0110] S1 Training Data Acquisition and Preprocessing

[0111] S1.1 Data Acquisition and Time Stamping

[0112] Data source: High-density electromyography acquisition system (electrodes consisting of 64 contacts arranged in an 8×8 matrix, with a distance of 10 mm between electrodes), force sensor (simultaneously using an A / D converter to digitize the force sensor signal, which has a 16-bit amplitude resolution, and the recognition system is shown in Figure 2).

[0113] Subjects and gestures: 10 healthy adults without forearm motor injury; target gestures were 65 common gestures (including single joint, two-finger movements, joint flexion and extension combinations, and multi-joint hand movements);

[0114] Training environment: Intel Core i7-12700K processor, NVIDIA RTX 4090 graphics card;

[0115] S1.2 Bandpass and Notch Filtering

[0116] Bandpass filtering: A 30Hz-500Hz bandpass filter is used.

[0117] Notch filtering: A notch filter with a center frequency of 50Hz is used.

[0118] S1.3 Electrode Interference Detection and Damaged Electrode Repair

[0119] The electrode interference detection method is as follows: when the amplitude of a certain signal segment exceeds a set threshold η, it is marked as interference: max(|x i [n]|)>ησ i

[0120] Where, x i [n] represents the signal amplitude, σ i This represents the standard deviation of the current segment signal.

[0121] The method for repairing damaged electrodes is through neighborhood bilinear interpolation:

[0122] in, For the channel being repaired, x (u,v) [n] represents the nearest passage, w uv To fix the weights.

[0123] S1.4 Standardization: Ensures consistent data distribution.

[0124] Specifically:

[0125] S1.4.1 Calculate the training set statistics, including the mean and variance.

[0126] S1.4.2. Signal Standardization Processing Based on Mean and Variance

[0127] S1.4.3. Save the obtained mean and variance to prepare for the reuse of test data.

[0128] Construction of spatiotemporal features of S2 electromyographic signals

[0129] S2.1 Electromyography Array Interpolation Resampling and Boundary Removal

[0130] Interpolation resampling: 8*8 channels, expanded to 18*18 channels using bilinear interpolation.

[0131] Boundary removal: Remove the outermost channel of the array and retain the middle 16*16 valid channels.

[0132] S2.2 Spatiotemporal gradient calculation and spatiotemporal feature construction.

[0133] The calculation process is shown in Figure 3, and the specific steps are as follows:

[0134] S2.2.1, 3D spatiotemporal gradient calculation, the specific calculation formula is as follows:

[0135] (a) Horizontal gradient G x (i,j,t)=EMG′(i+1,j,t)-EMG′(i-1,j,t)

[0136] (b) Vertical gradient G y (i,j,t)=EMG′(i,j+1,t)-EMG′(i,j-1,t)

[0137] (c) Multiscale temporal (causal) difference

[0138] Among them, different Δt k Used to capture changes in electromyographic signals at different time scales

[0139] (d) Synthesize the three-dimensional gradient vector at each position (i,j,t):

[0140] Where η is a scaling factor that balances the spatial and temporal dimensions.

[0141] (e) Calculate the gradient magnitude:

[0142] (f) Spatial orientation angle θ(i,j,t) (plane angle):

[0143] (g) Time tilt angle φ(i,j,t) (angle of spatiotemporal variation):

[0144] S2.2.2. Construction of 3D Spatiotemporal Features

[0145] (a) Divide the current frame into n subframes according to certain rules. The following steps (b) and (c) are performed in the subframes. (d) Stitch all the subframes together to obtain the three-dimensional spatiotemporal features.

[0146] This implementation plan is fixed according to the channel location (dividing the 16*16 channel topology into 16 4*4 channel topologies).

[0147] (b) Discretize θ(i,j,t) and φ(i,j,t) in two dimensions: Discretize the spatial orientation angle θ(i,j,t) into K θ K θ =8) interval; discretize the time tilt angle φ(i,j,t) into K φ K φ =4) Intervals. For each time t, calculate the sum of the amplitudes within each interval:

[0148] Wherein, H(k) θ ,k φ (t) represents the spatiotemporal orientation distribution within the current time frame, with an output dimension of K. θ ×K φ =32.

[0149] (c) Subframe spatiotemporal feature construction: The statistical results of all times are concatenated in chronological order to obtain the final shape [T,K]. θ ×K φ The spatiotemporal feature block of the m-th subframe: f m =[H(·,·,1),H(·,·,2),…,H(·,·,T)]

[0150] (d) Construction of 3D spatiotemporal features: The statistical results of all subframes (m = 1, 2, ..., n) are concatenated according to the splitting order in (a) to obtain the final 3D spatiotemporal features:

[0151] The process of building and training the S3 joint model is shown in Figure 4.

[0152] S3.1 Phase-Physiological Joint Temporal Encoding, the specific calculation process is as follows:

[0153] A phase encoding φ t :

[0154] The clip function will Mapped to the interval (0, 1), t onset T is the start time of the detection action. win This represents the total transient time range.

[0155] B-force perception weight w t

[0156] in, These are frame-by-frame force prediction values, where ∈ represents a small amount required for normalization to improve numerical stability. C yields the final joint temporal coding.

[0157] S3.2 Modeling of the selective attention mechanism guided by frame-by-frame force signals, the specific steps are as follows:

[0158] S3.2.1 The spatiotemporal features obtained from s2.2 Joint time coding obtained with s3.1 By splicing the data along the time dimension, we obtain the spatiotemporal fusion feature X. t Subsequent operations are based on the spatiotemporal fusion feature X.t Expand.

[0159] S3.2.2 Calculate Q, K, and V values, and calculate the input spatiotemporal fusion feature X. t The result is obtained through three sets of linear transformations:

[0160] Q = X t W Q K = X t W K V = X t W V

[0161] Among them, W Q W K W V These are the learnable parameters for the query vector Q, key vector K, and value vector V, which are randomly initialized during training.

[0162] S3.2.3 Constructing a time-weighted square matrix M based on predictive power

[0163] Among them, M ij σ is the element in the i-th row and j-th column of the square matrix M, representing the difference between the i-th and j-th frames in the "force signal" dimension, and σ(·) represents the normalization function.

[0164] S3.2.4 Improved attention calculation formula:

[0165] Where d k Let K be the feature dimension of Q in the attention calculation process, and ρ be the control force signal constraint strength S3.2.5 output feature: Z = AV

[0166] Where A is the attention weight matrix calculated in S3.2.4, and V is the value vector calculated in S3.2.2.

[0167] S3.2.6 Force Signal Regression Branch

[0168] After the feature vector of each frame, a linear regression layer is used to output the corresponding force prediction value: f t =W force Z t +b force

[0169] Among them, W force Let b be the weight matrix. force For bias terms,

[0170] S3.2.7 Gesture Classification Branch

[0171] First, perform intra-frame average pooling, then perform cross-frame fully connected layers, and finally use the output gesture prediction value:

[0172] (a) Intra-frame average pooling:

[0173] Among them, Z t Let h represent the features of frame t, h·w represent the feature dimension of each frame, and s represent the features of frame t. t The features are the pooled features for each frame.

[0174] (b) Frame by frame stitched into a vector:

[0175] Where T represents the total time dimension.

[0176] (c) Fully connected mapping and probability output: o = W c s+b c

[0177] Among them, W c Let b be the weight matrix of the fully connected network. c For classification bias, o represents the unnormalized score of each category, C represents the total number of categories, and o c To predict the unnormalized score for class c, To predict the probability of being in class c. S3.3 Force signal-based supervised training, the total joint loss function should be based on the basic classification crossover loss L. class Above this, increase the force regression constraint loss L force and physiological and dynamic constraint loss L physio :

[0178] S3.3.1 Cross-loss constraint for classification task

[0179] Where C represents the total number of categories, and y c It is the one-hot encoding of the real label. It is the probability that the model predicts it to be class c.

[0180] S3.3.2 Force Regression Constraint:

[0181] Where T is the total number of time frames. For true strength, For predictive power,

[0182] S3.3.3 Physiological and dynamic constraints,

[0183] To satisfy the law of force change during muscle contraction, three constraints are designed for force prediction in each frame:

[0184] g) Use "smoothness constraint" to constrain the drastic changes between two adjacent frames:

[0185] h) Use "monotonicity constraint" to constrain the change trend between adjacent frames:

[0186] i) Use "rate limit constraint" to constrain the rate of change between adjacent frames:

[0187] Where T is the total number of time frames. The predicted value of the force at time t-1. R is the predicted force value at time t. max The physiological upper limit of force change between two adjacent frames

[0188] Therefore, the total physiological dynamics constraint loss should simultaneously consider the above three constraint losses, and is defined as: L physio =λ s L smooth +λ m L mono +λ r L rate

[0189] Where, λ s , λ m , λ r These are the weights of the three constraints.

[0190] In this example, a total of 5 channels of force sensors were used, therefore there are 5 independent L-terms. force and five independent L physio The calculation approach for each item is exactly the same as the above process. To avoid ambiguity, the force regression constraint loss of the h-th channel is denoted as... The physiological dynamic constraint loss of the h-th channel is denoted as

[0191] S3.3.4 The overall joint loss function is given by the basic classification crossover loss L. class Above this, increase the force regression constraint loss L force and physiological and dynamic constraint loss L physio :

[0192] Where, α, β h γ h (h = 1, 2, ..., 5) represents the weights of the three types of losses.

[0193] S4.1 Test Data Preprocessing

[0194] Test scenarios: Divided into same-subject testing and transfer testing. The former uses data from the same person for training and testing, while the latter uses data from nine other people for training and data from another person for testing.

[0195] Data Acquisition and Preprocessing: Standardized Steps and Reuse of Training Set Statistics μ train With σ train Standardize the electromyographic signals during the testing phase to avoid data distribution deviation.

[0196] S4.2 Spatiotemporal Feature Extraction:

[0197] Reuse the training process to obtain spatiotemporal features

[0198] S4.3 Gesture Classification Based on the Model During Training

[0199] Only electromyographic information is used. Based on the spatiotemporal feature extraction of s2.2 and the phase-physiological joint temporal encoding of s3.1, the spatiotemporal fusion feature X is obtained. t The model uses a selective attention mechanism guided by frame-by-frame force signals from input S3.2 to obtain gesture classification results. Simultaneously, it outputs force information frame-by-frame as a reference, improving the model's interpretability.

[0200] Experimental results:

[0201] Improved accuracy: By using the transient phase of electromyographic signals for recognition of multiple gestures, the accuracy rate is improved to 84.6%, which is 17.4% higher than the traditional single feature method (67.2%). All data comes from before the hand movement is completed, thus meeting the real-time response requirements of electromyographic prostheses.

[0202] The generalization ability is significantly enhanced: the recognition accuracy in cross-subject scenarios reaches 73.8%, which is 11.3% higher than the unsupervised model (accuracy 62.5%), and can adapt to rehabilitation equipment scenarios shared by multiple users.

[0203] It should be noted that the above embodiments are not intended to limit the scope of protection of the present invention. Equivalent transformations or substitutions made based on the above technical solutions all fall within the scope of protection of the claims of the present invention.

Claims

1. A method for rapid gesture recognition by fusing the spatiotemporal characteristics of transient electromyographic signals with force sensor signals, characterized in that, The method includes the following steps: S1 Training Data Acquisition and Preprocessing includes: S1.1 Data Acquisition and Time Stamping, S1.2 Bandpass and Notch Filtering, S1.3 Electrode Interference Detection and Damaged Electrode Repair, S1.4 Normalization. S2 Spatiotemporal Feature Construction of EMG Signals, including: S2.1 EMG array interpolation resampling and boundary removal, S2.2 Spatiotemporal gradient calculation and spatiotemporal feature construction. The S3 joint model construction and training includes: S3.1 phase-physiological joint temporal encoding, S3.2 frame-by-frame force signal-guided selective attention mechanism modeling, and S3.3 force signal-based supervised training. The S4 testing phase and real-time recognition include: S4.1 test data preprocessing, S4.2 spatiotemporal feature extraction, and S4.3 gesture classification based on the training phase model.

2. The method for rapid gesture recognition by fusing the spatiotemporal characteristics of transient electromyographic signals with force sensor signals according to claim 1, characterized in that, In step S1.3, during electrode interference detection and damaged electrode repair, when the signal amplitude of a certain channel exceeds a set threshold ε, it is marked as electrode interference: max(|x i [n]|)>εσ i Where, x i [n] represents the signal amplitude, σ i The standard deviation of the current segment signal. Damaged electrode repair using neighborhood bilinear interpolation: in, For the repaired channel, x (u,v) [n] represents the nearest passage, w uv To fix the weights.

3. The method for rapid gesture recognition by fusing the spatiotemporal characteristics of transient electromyographic signals with force sensor signals according to claim 1, characterized in that, In S2.2, two-dimensional HOG feature calculation is performed in the spatial dimension, and causal difference is performed in the temporal dimension to calculate the time tilt angle, thereby extracting spatiotemporal features and multi-scale temporal (causal) difference. The calculation formula is: Among them, different Δt k Used to capture changes in electromyographic signals at different time scales The formula for calculating the time tilt angle φ(i,j,t) (the angle of spatiotemporal change) is: Among them, G x G y G t These represent the horizontal gradient, vertical gradient, and temporal (causal) difference, respectively, while η represents the scaling factor that balances the spatial and temporal dimensions.

4. The method for rapid gesture recognition by fusing the spatiotemporal characteristics of transient electromyographic signals with force sensor signals according to claim 1, characterized in that, In S3.1, in time-based phase encoding φ t Based on this, a frame-by-frame force prediction value is introduced. Force perception weight w t This yields the final joint time code. The specific calculation process is as follows: A phase encoding φ t : The clip function will Mapped to the interval (0, 1), t onset T is the start time of the detection action. win For the total transient time range, B-force perception weight w t in, These are frame-by-frame force predictions, where ∈ represents a small amount needed for normalization to improve numerical stability. C obtains the final joint time code.

5. The method for rapid gesture recognition by fusing the spatiotemporal characteristics of transient electromyographic signals with force sensor signals according to claim 1, characterized in that, In S3.2, a time-weighted matrix M based on predictive power is constructed, and the attention mechanism formula is improved. The formula for calculating the time-weighted square matrix M based on predictive power is: in, m represents the force prediction values ​​for frames i and j. ij is the element in the i-th row and j-th column of the square matrix M, representing the difference between the i-th and j-th frames in the "force signal" dimension, and σ(·) represents the normalization function. The improved attention calculation formula is as follows: Where d k Let Q be the query vector in the attention calculation process, K be the feature dimension of the key vector, and ρ be the constraint strength of the control force signal.

6. The method for rapid gesture recognition by fusing the spatiotemporal characteristics of transient electromyographic signals with force sensor signals according to claim 1, characterized in that, In S3.3 force signal-based assisted supervised training, in order to satisfy the law of force change during muscle contraction, a physiological dynamic constraint loss L is introduced. physio : a) Use "smoothness constraint" to constrain the drastic changes between two adjacent frames: b) Use "monotonicity constraint" to constrain the change trend between adjacent frames: c) Use "rate limit constraint" to constrain the rate of change between adjacent frames: Where T is the total number of time frames. The predicted value of the force at time t-1. R is the predicted force value at time t. max This represents the physiological upper limit of force change between two adjacent frames.

7. A rapid gesture recognition system that fuses the spatiotemporal characteristics of transient electromyographic signals with force sensor signals, characterized in that, This method is used to implement the rapid gesture recognition method that fuses the spatiotemporal features of transient electromyography signals with force sensor signals, as described in any one of claims 1-6. The system includes: Electromyography (EMG) acquisition unit: includes high-density EMG signal acquisition electrodes and a signal amplifier. Force signal acquisition unit: Acquires force signals from finger positions during gesture execution. Synchronization trigger unit: synchronizes the electromyography acquisition unit and the force signal acquisition unit in time. Processing unit and storage unit: These units perform instruction control, signal processing, and signal storage. Display unit: Displays the current command, real-time force signal, and electromyographic signal.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements a rapid gesture recognition method that fuses the spatiotemporal features of transient electromyographic signals with force sensor signals, as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the computer instruction is executed by the processor, it implements a rapid gesture recognition method that fuses the spatiotemporal characteristics of transient electromyographic signals with force sensor signals as described in any one of claims 1-7.