Web Resource Detection Using Token Vector Classifiers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for detecting harmful web content on networks are inadequate due to reliance on manually compiled lists of URLs and keywords, which fail to account for new resources and can lead to false blocking, and lack accuracy in identifying harmful content.
Innovation Solution
A method and system that uses a specifically trained classifier to analyze web resources by converting HTML documents into token vectors, applying natural language processing techniques, and determining a likelihood parameter for harmful content, allowing for improved classification without reliance on predetermined lists or keyword sensitivity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If manually compiled lists of URLs and keywords are used for filtering harmful content, then the filtering process is simple to implement, but the detection accuracy is low and false blocking occurs
Solution Approach 1:
The patent replaces manual keyword-based filtering with an automated machine learning classification system. The classifier is trained on labeled datasets to automatically distinguish harmful from benign content, substituting the mechanical keyword-matching approach with an intelligent system that achieves higher accuracy while reducing false positives.
Solution Approach 2:
The system transforms the detection approach by changing from static keyword lists to dynamic probabilistic classifications. The classifier outputs likelihood scores that can be adjusted via thresholds, allowing flexible control over detection sensitivity and false positive rates, thereby improving both accuracy and adaptability.
2Ease of manufacture
If manually compiled lists of URLs are used for filtering, then the implementation is straightforward, but the system cannot account for newly appeared resources
Solution Approach 1:
The classification system is designed to be self-updating through continuous training on new labeled data. As new harmful resources appear, they can be added to the training dataset and the classifier retrained, enabling the system to automatically adapt to emerging threats without manual list updates.
Solution Approach 2:
The system transitions from static keyword lists to a dynamic machine learning model that continuously learns from new data. The classifier's ability to be retrained on evolving datasets makes it adaptable to new harmful content patterns, domains, and techniques that emerge over time.
3Device complexity
If predetermined keywords are used for recognition, then the filtering process is simple, but false blocking occurs when keywords are used in figurative sense
Solution Approach 1:
The patent replaces rigid keyword-matching mechanics with flexible semantic understanding through machine learning. The classifier analyzes contextual patterns, word relationships, and content structure to distinguish literal from figurative usage, dramatically reducing false blocking while maintaining simple operation for end users.
4Measurement precision
If trained classifiers are used to analyze web resources, then the detection accuracy is improved, but the system complexity increases
Solution Approach 1:
The patent extracts and separates the complex training and model development processes from the operational filtering system. The classifier is trained offline on labeled datasets, and only the trained model is deployed for actual content filtering. This extraction allows high accuracy detection while keeping the operational system relatively simple.
Solution Approach 2:
The system performs the complex work of training and pattern recognition in advance during the offline training phase. By preparing the classifier model beforehand with pre-processed training data, the system eliminates the need for complex real-time analysis during actual content filtering, reducing operational complexity while maintaining high detection accuracy.
Data Source
AI summary
A method and a system for detecting harmful content on a network are provided. The method comprises: receiving a URL; obtaining, from the URL, an HTML document associated therewith; converting the HTML document into a text; normalizing the text associated with the HTML document, thereby generating a plurality of tokens associated therewith; aggregating, each one of the plurality of tokens into a token vector associated with the HTML document; and applying, one or more classifiers to the token vector associated with the HTML document to determine a likelihood parameter indicative of the URL being associated with the harmful content; in response to the likelihood parameter being equal to or greater than a predetermined likelihood parameter threshold: identifying, the URL as being associated with the harmful content; and storing, the URL in a database of harmful URLs.