Web Server Scanning Detection via URI Regularization

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional detection techniques for web server scanning activities are prone to erroneous detection and fail to accurately identify scanning targets, particularly when attackers adjust access patterns or use commonly used URIs.

Innovation Solution

A detection device that standardizes Uniform Resource Identifiers (URIs) in access logs, calculates relative frequencies of access logs corresponding to specific response codes, and determines scanning targets based on a threshold score, enabling accurate detection of scanning activities.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If signature matching technique is used to detect scanning, then detection can be performed for known scanning patterns, but scanning that has not been registered as a signature cannot be detected

Engineering Contradiction:

Improvedetection accuracy for known scanningVSAvoiddetection capability for unknown scanning

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The system pre-calculates and stores relative frequency thresholds for different response codes before actual scanning detection occurs. This preliminary preparation enables the system to quickly compare actual access patterns against pre-established benchmarks, allowing detection of both known and unknown scanning activities without requiring pre-registration of specific scanning signatures

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system transitions from static signature matching to dynamic parameter-based detection by calculating relative frequencies of response codes and comparing them against threshold values. This parameter change enables the system to detect various scanning patterns based on statistical anomalies rather than predetermined signatures, improving adaptability to unknown scanning techniques

Inventive Principle:

Principle #35Parameter changes

2Measurement precision

If detection is based on HTTP response codes (e.g., 404), then scanning can be detected, but normal web accesses resulting in errors are erroneously detected as scanning

Engineering Contradiction:

Improvescanning detection capabilityVSAvoiderroneous detection rate

Core Design Contradiction:

Measurement precisionVSReliability

Solution Approach 1:

Instead of detecting scanning based on a single response code (exact detection), the system calculates relative frequencies of multiple response codes and compares them against threshold values. This partial action approach allows the system to tolerate some erroneous detections in exchange for significantly reduced false positives, as normal error-prone accesses typically don't exhibit the same frequency patterns as systematic scanning

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The system uses relative frequency calculations as feedback to distinguish between scanning and normal access. By continuously monitoring and comparing the frequency of response codes against established thresholds, the system can adaptively identify patterns characteristic of scanning behavior while filtering out normal error-based accesses

Inventive Principle:

Principle #23Feedback

3Measurement precision

If detection is based on co-occurrence of URIs, then scanning targets can be identified, but commonly used URIs such as '/' and '/index.php' are erroneously detected

Engineering Contradiction:

Improvescanning target identificationVSAvoidfalse positive rate for common URIs

Core Design Contradiction:

Measurement precisionVSReliability

Solution Approach 1:

The system applies different detection criteria to different URIs based on their local characteristics. Rather than uniformly detecting all co-occurring URIs, it calculates relative frequencies specific to each URI and compares them against customized threshold values, allowing common URIs to be excluded from erroneous detection while maintaining detection of actual scanning targets

Inventive Principle:

Principle #3Local quality

4Reliability

If conventional detection techniques are used, then scanning can be detected, but the techniques fail when attackers adjust access patterns or extend intervals

Engineering Contradiction:

Improvedetection effectivenessVSAvoidresistance to pattern adjustment

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The system transitions from static detection rules to dynamic relative frequency analysis. By continuously calculating and comparing frequencies of response codes against threshold values, the system can adapt to changing access patterns in real-time, making it effective against attackers who adjust their scanning intervals or patterns

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS11818153B2Detection device and detection program

Publication Date: 2023.11.14 NIPPON TELEGRAPH & TELEPHONE CORP

AI summary

A regularization unit standardizes similar expressions across a plurality of URIs in access logs of requests made to a plurality of web servers, thereby changing the URIs into regularized URIs. A calculation unit calculates, among the access logs that are from the same source, the relative frequency of certain access logs to all access logs, the certain access logs corresponding to requests made to different destinations for the same regularized URI and also corresponding to certain response codes. If the largest of all the relative frequencies calculated for the regularized URIs is at least a certain threshold, a determination unit determines the regularized URIs to be scanning targets.