Malware Detection via Dynamic Crawler Feature Extraction

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

The architecture of malware distribution networks (MDNs) makes it difficult for search engines to identify infected landing pages, redirect servers, and exploit servers due to the inability of static crawlers to execute scripts, rendering malicious actions invisible and challenging the detection of malicious web pages.

Innovation Solution

A system that employs a dynamic crawler to analyze web pages, execute scripts, and follow links, allowing for the identification of landing pages, redirect servers, and exploit servers, which are then used to learn features indicative of MDNs. These features are extracted by a static crawler and employed to label web pages as belonging to an MDN, enabling their removal from search engine results and indexes.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If a static crawler is used to index web pages, then the crawling speed and coverage are improved, but the ability to detect malicious scripts is lost

Engineering Contradiction:

Improvecrawling speedVSAvoidmalware detection accuracy

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system divides the web crawling and analysis task into two segments: a static crawler that rapidly indexes web pages for coverage, and a separate dynamic analysis system that selectively executes scripts on suspicious pages. This segmentation allows the static crawler to maintain high productivity while the dynamic system ensures reliability by detecting malicious scripts that would otherwise remain invisible.

Inventive Principle:

Principle #1Segmentation

2Reliability

If a dynamic crawler is used to execute scripts, then the detection of malicious actions is improved, but the crawling speed and efficiency decrease

Engineering Contradiction:

Improvemalware detection accuracyVSAvoidcrawling speed

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

Instead of having all crawlers execute scripts (excessive action), the system applies partial action by having only the dynamic crawler execute scripts on a selective basis. The static crawler handles the majority of web pages without script execution, while the dynamic crawler performs targeted script execution on suspicious pages identified by the static crawler or through sampling, thus maintaining overall system productivity while improving detection reliability.

Inventive Principle:

Principle #16Partial or excessive action

3Area of stationary object

If the number of web pages to be indexed is increased, then the search engine coverage is improved, but the ability to analyze each page for malicious content is reduced

Engineering Contradiction:

Improvesearch engine coverageVSAvoidmalicious content analysis accuracy

Core Design Contradiction:

Area of stationary objectVSMeasurement precision

Solution Approach 1:

The system applies different quality levels of analysis to different web pages based on their risk profile. High-coverage indexing is applied universally to maintain search engine coverage, while deep malicious content analysis with script execution is applied locally only to suspicious pages identified through various indicators. This local quality approach ensures comprehensive coverage while maintaining high detection precision for malicious content.

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS10885190B2Identifying web pages in malware distribution networks

Publication Date: 2021.01.05 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

Technologies pertaining to analyzing content extracted from web pages by a static crawler to determine whether respective web pages are members of a malware distribution network (MDN) are described. A set of features is learned based upon output of a dynamic crawler over known landing pages of a particular MDN, wherein the set of features are indicative of membership in the MDN. Using such set of features, additional members of the MDN (not subjected to crawling by a dynamic crawler) are identified.