Asset Crawling Through Internet Archives for Web Vulnerability Detection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional Internet crawlers fail to detect vulnerabilities in web applications when functionality is removed or hidden on the front end, leading to potential security risks due to overlooked URLs that still function on the back end.

Innovation Solution

A system that utilizes Internet archives to retrieve snapshots of dynamic pages no longer available on the front end, scans these pages for vulnerabilities, and takes security actions when vulnerabilities are identified.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If conventional active crawlers are used to scan web applications, then scanning efficiency is improved, but vulnerabilities in removed or hidden functionalities are overlooked

Engineering Contradiction:

Improvescanning efficiencyVSAvoidvulnerability detection accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The system performs preliminary actions by querying Internet archives for historical URLs before conducting active scanning. This allows the crawler to identify and scan removed or hidden functionalities that conventional crawlers would miss, thereby improving vulnerability detection accuracy without significantly compromising scanning efficiency

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

Internet archives serve as an intermediary between the crawler and the target web application. By querying archives for historical URLs and using them as a basis for scanning, the system bridges the gap between active scanning efficiency and comprehensive vulnerability detection, including functionalities that have been removed or hidden from the current front end

Inventive Principle:

Principle #24Intermediary (Mediator)

2Use of energy by moving object

If the crawler focuses only on currently available pages, then resource consumption is reduced, but the attack surface is underestimated due to missed inactive URLs

Engineering Contradiction:

Improvecrawler resource consumptionVSAvoidattack surface coverage

Core Design Contradiction:

Use of energy by moving objectVSObject-affected harmful factors

Solution Approach 1:

The system performs preliminary queries to Internet archives to obtain historical URL information before conducting active scanning. This allows the crawler to identify and include removed or hidden functionalities in its scanning scope, thereby expanding attack surface coverage without requiring continuous monitoring of all possible URLs

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

Internet archives act as an intermediary that provides historical URL data, enabling the crawler to expand its scanning scope to include inactive URLs. This approach allows comprehensive attack surface coverage while maintaining reasonable resource consumption by leveraging pre-existing archive data rather than continuously discovering all URLs

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS20250301009A1Asset crawling with internet archives for enhanced web application scanning

Publication Date: 2025.09.25 FORTINET INC

AI summary

A scan request for a domain includes at least some dynamic pages that are no longer available on a front end of a web host. Responsive to not being available on the front end of the web host, a list of URLs is retrieved from an archive server that stores snapshots of the dynamic pages from when they were available on the front of the web host. The list of retrieved URLs is examined, with a back end of the web host, for vulnerabilities. Responsive to identifying at least one vulnerability on at least one of the dynamic pages, a security action is taken with respect to the at least one dynamic page.