The invention provides an interrogation
record and audio and
video recording linkage method and
system, and belongs to the technical field of
information processing, and the method comprises the steps: obtaining interrogation
record text data and segmented
video recording data, and respectively labeling category labels; performing semantic extraction on the
record text data to obtain
semantic feature information, and converting the feature information into text
semantic feature vectors; frame extraction is carried out on the video data, image features of each frame of the video data are extracted, similarity screening is carried out, and the image features are converted into
key frame semantic feature vectors; constructing a cross-
modal embedding
space model, respectively encoding the text semantic
feature vector and the
key frame semantic
feature vector of the category
label through two encoders, then splicing and fusing to obtain a fused
feature vector, converting and mapping the fused feature vector into a sharing layer, completing feature association of the record text data and the segmented
video recording data, and obtaining the segmented video recording data. And subsequently, cross-
modal search is carried out. According to the method, cross-
modal information association is realized through association of the semantic feature vectors of the record and video recording information, and a basis is provided for subsequent rapid calling, research and judgment.