Unsupervised URL Anomaly Detection via One-Hot Encoding and Autoencoders
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional methods for detecting abnormal URLs rely on fixed rules, which are inadequate for identifying unknown characteristic URLs, leading to difficulties in accurately distinguishing normal from abnormal URLs, thereby increasing the risk of malicious access.
Innovation Solution
An unsupervised exception access detection method utilizing a bi-gram model, one-hot encoding mechanism, and deep autoencoder network with cluster analysis to identify abnormal URLs, enabling accurate detection even when characteristics are unknown and abnormal samples are scarce, and allowing for offline training with fast detection speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If fixed rules are used for URL detection, then known abnormal URLs can be identified, but unknown characteristic abnormal URLs cannot be accurately detected
Solution Approach 1:
The patent transforms URL detection from rule-based matching to a statistical parameter-based approach. It extracts multiple features from URLs including character frequency, word frequency, path depth, query parameters, and structural characteristics. These parameters are then used to build a comprehensive detection model that can identify both known and unknown abnormal URLs through statistical analysis rather than fixed pattern matching.
Solution Approach 2:
The patent replaces the mechanical rule-based detection system with a machine learning-based statistical system. Instead of manually defined detection rules, it uses algorithms to automatically learn detection patterns from training data. The system employs techniques such as Naive Bayes classification, support vector machines, and neural networks to substitute the rigid mechanical rule system with a flexible intelligent detection mechanism.
2Measurement precision
If machine learning methods are used to detect abnormal URLs, then detection accuracy improves, but detection complexity and computational resources increase
Solution Approach 1:
The patent divides the URL detection system into multiple independent modules: feature extraction module, model training module, and detection module. Each module handles specific tasks separately. The feature extraction breaks down URLs into character-level, word-level, and structural features. The detection system is segmented into multiple algorithms that can be independently optimized and combined, reducing overall system complexity while maintaining high accuracy.
Solution Approach 2:
The patent implements an offline training phase where the detection model is pre-trained on a large corpus of normal and abnormal URLs before actual detection. This preliminary action allows the system to learn detection patterns in advance, so that during runtime, only lightweight feature extraction and classification are needed. The complex model training is performed beforehand, making the actual detection process simpler and faster.
3Measurement precision
If comprehensive feature extraction is performed on URLs, then detection accuracy improves, but processing time and computational cost increase
Solution Approach 1:
The patent applies different levels of feature extraction to different parts of the URL based on their diagnostic value. Not all URL components are analyzed with the same depth. For example, the domain name part may receive more detailed analysis than the query parameters. The system dynamically adjusts the extraction intensity for different URL segments, focusing computational resources on the most informative local regions rather than uniformly processing the entire URL.
Solution Approach 2:
The patent implements a multi-stage detection process where not all features are extracted and analyzed for every URL. In the first stage, quick filters are applied to obviously normal or abnormal URLs. Only URLs that require further analysis undergo complete feature extraction. This partial action approach processes fewer URLs with full complexity, reducing average processing time while maintaining detection accuracy for suspicious URLs.
Data Source
AI summary
The present disclosure provides an unsupervised exception access detection method and apparatus based on one-hot encoding mechanism. The method includes: encoding each test URL sample by using one-hot encoding mechanism, to obtain a high-dimensional vector; inputting the high-dimensional vector into a pre-built deep autoencoder network for compression and dimension reduction processing, to obtain a two-dimensional vector; performing visualization operation on the two-dimensional vectors by using a two-dimensional coordinate system, to obtain visualized test URL samples; performing a cluster analysis on all visualized test URL samples by using a K-means algorithm, to divide the test URL sample set into a first type and a second type of URL sets; comparing sample sizes of the first type and second type of URL sets, determining the URL set with a larger sample size as a normal URL set, and determining the URL set with a smaller sample size as an abnormal URL set.