URL Processing Apparatus for Web Vulnerability Scanning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Web crawling collects numerous duplicated or useless URLs, leading to inefficient work and increased server load during web vulnerability scanning, as existing methods lack an effective scheme to filter out unnecessary URLs based on purpose.
Innovation Solution
A URL processing method that determines valid URLs by using header information to exclude irrelevant URLs and calculates similarity or shared paths among collected URLs, selecting only those that are relevant for specific purposes, such as web vulnerability scanning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If web crawling collects all URLs from a web site, then the quantity of collected URLs increases, but the proportion of duplicated or useless URLs increases leading to inefficient work
Solution Approach 1:
The patent applies preliminary action by performing URL filtering and validation operations during the web crawling process itself, rather than after collection. The URL processing apparatus determines whether to exclude URLs from the valid URL list using header information before the crawling completes, preventing useless URLs from being processed further and thereby maintaining high work efficiency while collecting comprehensive URLs
Solution Approach 2:
The patent extracts and removes duplicated or useless URLs from the collected URL set through a filtering mechanism. The URL processing apparatus identifies and excludes unnecessary URLs based on header information analysis, separating valid URLs from invalid ones, thus improving productivity by ensuring only useful URLs proceed to subsequent processing stages
2Quantity of substance
If web vulnerability scanning is performed on all collected URLs including image files, then comprehensive scanning coverage is achieved, but the usefulness and efficiency of the scanning decreases
Solution Approach 1:
The patent applies local quality by differentiating the treatment of different URL types based on their characteristics. The URL processing apparatus analyzes header information to determine the nature of each URL (e.g., HTML page vs. image file) and applies appropriate filtering rules specifically tailored to each type, ensuring that web vulnerability scanning is performed only on relevant URL types while maintaining comprehensive coverage where applicable
Solution Approach 2:
The patent performs preliminary classification and filtering of URLs by type before web vulnerability scanning. The URL processing apparatus uses header information to identify and exclude irrelevant URL types (such as image files) from the valid URL list prior to scanning, ensuring that scanning resources are allocated only to URLs that can yield meaningful vulnerability assessment results
3Quantity of substance
If multiple URLs with the same or similar web page style are collected, then comprehensive collection is achieved, but server load increases due to duplicative scanning
Solution Approach 1:
The patent extracts and removes duplicate URLs from the collected set by comparing web page styles and content. The URL processing apparatus identifies URLs with identical or similar styles through header information analysis and excludes duplicates from the valid URL list, thereby reducing server load from duplicative scanning while maintaining comprehensive collection of unique URL types
Solution Approach 2:
The patent changes the parameter of URL selection criteria by incorporating header information analysis and web page style comparison. Instead of treating all URLs equally, the system evaluates URLs based on their stylistic parameters and content characteristics, selecting representative URLs for scanning while excluding duplicates, thus optimizing server resource utilization
Data Source
AI summary
A URL processing method includes a response data determining step in which a URL processing apparatus determines whether to exclude one or more URLs included in a first web page from a valid URL list using header information for the first web page of a first web site and a similarity based valid URL calculating step of estimating a similarity between web pages corresponding to respective URLs according to a predetermined criterion with respect to one or more URLs included in the first web page and selecting some of URLs of a similar web page calculated according to the similarity and adding the selected URLs in the valid URL list.