The invention discloses a semantic segmentation and
pose estimation method and
system for a
surgical robot instrument under an
endoscope image, and relates to the field of
computer vision. The objective of the invention is to solve the problems that an existing semantic segmentation and
pose estimation method of a
surgical robot instrument is large in error rate and poor in real-time performance and generalization performance, the constraint effect of
time sequence information is rarely considered, and the requirement of accurate
surgical operation is difficult to meet. According to the technical key points, a
backbone network is used for extracting multi-scale information of an original image of an
endoscope, and a feature encoding module aggregates encoded features from features extracted from the
backbone network and comprises image local feature information of the image at different abstraction levels; the semantic segmentation decoder is used for generating segmented images from the image feature map, and the segmented images provide implicit geometric constraints for the
pose estimation decoder; based on geometric constraints provided by a semantic segmentation result, the position and posture estimation decoder regresses the positions and posture parameters of the
wrist and the rod of the
surgical instrument relative to the camera from the feature map, and the opening and closing angle of the
tail end of the instrument; and a
time sequence module is used to construct a
time sequence data buffer queue for continuous frame analysis, inter-frame
jitter is reduced, and the precision of pose estimation and end opening and closing angle estimation algorithms is improved.