Webpage Similarity Analysis Using HTML Trees and Script Structure
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for identifying webpages and network assets face challenges in accurately determining fingerprint information, especially when metadata is incomplete or hidden, leading to difficulties in identifying complex webpages and managing network assets effectively.
Innovation Solution
A method that determines webpage similarity by combining HTML structure trees, script file structures, and metadata, using a weighting algorithm to calculate similarities, and identifies network assets by comparing webpage similarities to determine fingerprint information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If webpage identification is based solely on metadata analysis, then the identification process is simple and fast, but identification accuracy deteriorates when metadata is incomplete or hidden
Solution Approach 1:
The patent combines multiple identification dimensions (metadata, HTML structure tree, script file structure) into a unified identification system. By merging these three aspects, the system achieves higher accuracy without relying on any single dimension, thereby resolving the contradiction between accuracy and complexity.
Solution Approach 2:
The patent segments the webpage identification into three independent components: metadata analysis, HTML structure tree analysis, and script file structure analysis. Each component can be processed separately and then integrated, which reduces the overall complexity by breaking down the complex identification task into manageable segments.
2Measurement precision
If complete HTML structure tree is used for similarity calculation, then identification detail is high, but calculation time increases due to large amount of data
Solution Approach 1:
The patent extracts and removes the bottom-most layer of nodes from the HTML structure tree before performing similarity calculation. This extraction eliminates redundant detailed information that contributes little to identification accuracy while significantly reducing the data volume, thereby decreasing calculation time without losing essential identification details.
3Reliability
If multiple identification factors are combined, then identification robustness improves, but computational complexity increases
Solution Approach 1:
The patent assigns different weights to different identification factors (metadata, HTML structure tree, script file structure) based on their local importance and reliability. By applying local quality differentiation through weighted scoring, the system achieves robust identification while managing computational complexity by focusing resources on more reliable factors.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Provided are a method and apparatus for determining a similarity between webpages, a method and apparatus for identifying network assets, and an electronic device and a storage medium. The method for determining a similarity between webpages comprises: acquiring metadata of a first webpage, a hyper text markup language source code of the first webpage and a script file structure of the first webpage; determining a hyper text markup language structure tree of the first webpage on the basis of the hyper text markup language source code of the first webpage; determining a first similarity between the hyper text markup language structure tree of the first webpage and a hyper text markup language structure tree of a second webpage; determining a second similarity between the script file structure of the first webpage and a script file structure of the second webpage; determining a third similarity between the metadata of the first webpage and metadata of the second webpage; and determining a similarity between the first webpage and the second webpage on the basis of the first similarity, the second similarity and the third similarity.