Machine Learning URL Categorization via Segmented Classifiers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in accurately and efficiently categorizing URLs into appropriate categories for web security policies, due to the dynamic nature of websites, limited content for analysis, and class imbalance in training data.
Innovation Solution
A machine learning-based URL categorization system that employs a multi-step process to eliminate noise in training data, uses post-processing rules to refine classification results, and maintains both offline and inline URL classification mapping databases to optimize performance in production environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a machine learning classifier is trained to categorize URLs into detailed categories, then classification accuracy is improved, but the complexity of the system and computational resources required increase
Solution Approach 1:
The patent divides the URL categorization system into multiple independent classifiers, each trained to predict a specific category. Instead of using one complex classifier for all categories, the system segments the problem into multiple simpler classifiers that can be independently trained and executed, reducing overall system complexity while maintaining high classification accuracy across all categories.
Solution Approach 2:
The patent introduces a post-processing engine as an intermediary component that receives predictions from multiple classifiers and resolves conflicts or ambiguities. This mediator layer协调s the outputs from individual classifiers, ensuring accurate final categorization without requiring each classifier to handle the full complexity of all categories.
2Measurement precision
If comprehensive training data is used to improve model accuracy, then classification performance is improved, but the time and resources required for training increase
Solution Approach 1:
The patent segments the training process by training each classifier independently on relevant subsets of training data rather than training one large classifier on all data. This allows parallel training of multiple smaller models, significantly reducing total training time while maintaining comprehensive category coverage through the collection of specialized classifiers.
Solution Approach 2:
The patent performs preliminary actions by pre-training individual classifiers on their respective category datasets before deployment. This preliminary training of specialized classifiers allows the system to achieve high accuracy without requiring extensive real-time computation, as the heavy lifting is done during the offline training phase for each classifier.
3Measurement precision
If both offline and inline URL classification mapping databases are maintained, then classification coverage and accuracy are improved, but storage and memory resources are consumed
Solution Approach 1:
The patent segments the URL classification database into two distinct components: an offline database for comprehensive URL-category mappings and an inline database for frequently accessed or critical URLs. This segmentation allows the system to maintain high classification coverage by distributing data across different storage layers, optimizing the balance between comprehensive coverage and resource consumption.
Solution Approach 2:
The patent applies local quality by maintaining different database characteristics in different locations - the offline database stores comprehensive mappings with higher storage capacity, while the inline database maintains a smaller, faster-access subset of critical URLs. This local optimization ensures that frequently needed classifications are quickly accessible while less frequently used data resides in the larger offline storage.
Data Source
AI summary
Disclosed technology of training a third classifier to select between sensitive or non-sensitive categories for a webpage including both sensitive and non-sensitive content. The technology involves collecting webpages as a tentatively labeled dataset A and a dataset B, training a first classifier including at least a first sensitive category classifier and a first non-sensitive category classifier using the tentatively labeled dataset A, referring some dual labelled webpages generated by the first classifier to a curator to curate and resolve label conflict, receiving curated labels from the curator and updating dataset B with the curated labels, training a second classifier using the updated dataset B, referring some dual labelled webpages generated by the second classifier to a curator to curate and resolve label conflict, receiving curated labels from the curator and updating dataset A with the curated labels, and training the third classifier with updated dataset A and updated dataset B.