Website Crawling via Schema-Based Content Segmentation

The patent segments the website content into distinct sections (e.g., header, navigation, main content, footer) and applies different crawling strategies to each segment. By analyzing the structural schema of the website, the system prioritizes crawling of content sections that are most likely to contain vulnerable content, such as download areas or user input fields, while reducing or skipping crawling of less critical sections. This segmentation enables focused resource allocation and reduces overall crawling time while maintaining identification accuracy.

The patent applies local quality by differentiating the crawling approach based on the specific location and type of content within the website schema. Different weights and priorities are assigned to different content regions based on their potential vulnerability. For example, content areas related to file downloads, user uploads, or form submissions are assigned higher priority for crawling and analysis, while static decorative elements are assigned lower priority. This localized differentiation optimizes the balance between thoroughness and efficiency.

The patent divides the website into schema-defined segments and applies selective crawling based on the potential information value of each segment. By analyzing the website structure beforehand, the system identifies which segments are most likely to contain vulnerable content and allocates computational resources accordingly. This segmentation strategy reduces unnecessary processing of low-value content areas while maintaining focus on high-risk areas, thereby reducing overall computational resource consumption.

The patent implements partial action by crawling and analyzing only the portions of the website that are most likely to contain vulnerable content, rather than systematically crawling every element. The schema-based approach enables the system to identify and prioritize critical content areas such as download sections, upload fields, and user interaction areas, while skipping or reducing analysis of less critical areas. This partial crawling approach maintains sufficient detection capability while significantly reducing computational resource usage.

The patent applies preliminary action by first analyzing the website schema structure before executing the crawling process. The system pre-identifies the organizational structure of the website, including hierarchical relationships between content elements, and uses this pre-analyzed schema to guide the crawling process. This preliminary structural analysis enables the system to prioritize crawling paths and allocate resources more effectively, achieving both comprehensive coverage of critical areas and improved crawling efficiency without requiring exhaustive analysis of every website element.

The patent implements local quality by differentiating the crawling depth and intensity based on the local importance of different content areas as defined by the website schema. Critical content areas that are more likely to contain vulnerable content receive higher crawling priority and more thorough analysis, while less critical areas receive reduced attention. This localized differentiation maintains reliability for identifying vulnerable content while significantly improving overall crawling efficiency by avoiding uniform treatment of all content.