The invention provides a three-dimensional hand posture
estimation and
action recognition algorithm based on
deep learning, and the
algorithm comprises the steps: obtaining a public first person hand action
data set FPHA, carrying out the preprocessing, taking a video sample according to an experiment condition and a set proportion, and dividing the video sample into a
training set and a
test set; a three-dimensional hand posture
estimation and
action recognition model which is based on
deep learning and comprises a double-layer
transformer and a lightweight posture
smoothing network smopnet is established; inputting the processed first person hand action
data set training set into a posture
estimation and
action recognition model according to a set batch by utilizing a hierarchical
moving window thought to carry out
supervised training, and sending the preprocessed data into a pre-trained ResNet model to carry out shallow
feature extraction; the method comprises the following steps: firstly, carrying out two-dimensional hand
gesture recognition on the two-dimensional hand gesture, then sending the two-dimensional hand gesture to a first-layer
transformer, carrying out gesture estimation and smooth optimization by utilizing coded short-time inter-
frame time sequence information, fusing the two-dimensional hand gesture, operation object classification information and a feature map after the first-layer coded
time sequence information by a second-layer
transformer in a longer time span, and modeling a
semantic relationship between the gesture and the action so as to judge the type of the action. Through the method provided by the invention, three-dimensional hand posture estimation with higher precision and action type judgment with higher accuracy can be carried out on the first person hand action video.