The present application belongs to the technical field of
computer vision, and particularly relates to an open-vocabulary
object detection method based on multi-
modal fusion and dynamic expansion, aiming to solve the problems of insufficient robustness of visual features, static multi-
modal fusion, and poor flexibility of class expansion in the prior art. The method first acquires an image to be detected and text information containing class and scene description, extracts features through a visual
encoder and enhances them through a robust
noise reduction layer, and then generates a candidate detection frame through a full convolutional network, while extracting text features through a CLIP text
encoder. Subsequently, the candidate region features are extracted, dynamically fused through an adaptive multi-
modal fusion module to generate cross-modal joint features, and finally the detection frame is screened and the confidence is calculated according to the feature similarity. The model training fuses the InfoNCE loss and the cross-entropy loss to optimize the parameters, and the new class dynamic expansion can also be realized through the analysis of
natural language by a generative
language model. The present application significantly improves the robustness and accuracy of detection in complex scenes, optimizes the multi-modal fusion efficiency, greatly reduces the cost of new class expansion, and the model meets the real-time detection requirements after acceleration, has good
engineering deployment advantages, and can be widely applied in the fields of intelligent monitoring, automatic driving and the like.