The invention relates to a URL duplicate removal method and
system for dynamic website XSS
vulnerability detection, and the method comprises the steps: firstly, extracting URL features,
JavaScript behavior features, form structure features and DOM structure features of a website page; secondly, splicing the extracted features into a multi-dimensional
feature vector, and calculating the multi-dimensional
cosine similarity of any two URL pages by adopting a
cosine similarity algorithm; thirdly, determining an optimal similarity threshold value through a
bisection method in combination with artificial
label data or a website sample set; and finally, removing the URLs exceeding the threshold value based on the optimal similarity threshold value. According to the method, structural
feature extraction and a multi-dimensional similarity measurement mechanism are fused, so that good self-adaptability is achieved, the discrimination capability and
processing efficiency of a
Web crawler system can be effectively improved, the repeated capture proportion is reduced, and the overall performance of
vulnerability detection and
page analysis tasks is improved. Experimental results show that compared with a traditional method, the method has the advantages that repeated URLs can be recognized and removed more accurately, and the misjudgment rate is remarkably reduced.