Dark Web Document Classification Data Preprocessing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Collecting and preprocessing high-quality learning data for dark web document classification is challenging due to the unique characteristics of dark web documents, including many duplicates and identifiers that do not appear well on the surface web, leading to difficulties in crawling and labeling, which affects learning efficiency and increases unnecessary labeling costs.
Innovation Solution
A method involving a terminal that generates masked word sets from dark web documents, calculates similarity using Jaccard similarity, and collects or discards documents based on a specific threshold value to avoid duplicates and preprocess the data for training an AI model, utilizing an identifier table to convert words into higher-meaning identifiers and prevent overfitting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If general crawling and labeling methods are used on dark web documents, then the process can be completed, but the quality of learning data is poor due to duplicates and unique dark web characteristics
Solution Approach 1:
The patent applies preliminary action by performing masking and deduplication operations before the actual labeling process. Documents are pre-processed to mask identifiers and remove duplicates, so that the subsequent labeling process works with cleaned data, improving quality while reducing the complexity of the labeling task itself.
Solution Approach 2:
The patent introduces an intermediary processing step between crawling and labeling. A preprocessing module acts as an intermediary that masks identifiers and filters duplicates, transforming raw dark web documents into a format suitable for high-quality labeling without requiring complex manual intervention.
2Quantity of substance
If all crawled dark web documents are collected for training, then more data is available, but duplicates increase labeling costs and reduce learning efficiency
Solution Approach 1:
The patent extracts and removes duplicate documents from the crawled dataset before labeling. By identifying and taking out duplicates through similarity comparison, the system maintains a sufficient quantity of unique training data while eliminating redundant content that would waste labeling resources and reduce learning efficiency.
Solution Approach 2:
The patent changes the parameter of data uniqueness by introducing similarity thresholds and deduplication filters. Documents are compared using similarity metrics, and those exceeding a threshold are removed, transforming the dataset from containing duplicates to containing only unique documents, thereby improving labeling efficiency.
3Loss of information
If identifiers in dark web documents are left unmasked, then more information is preserved, but the AI model may overfit and classification performance decreases
Solution Approach 1:
The patent extracts and removes identifier information from dark web documents through masking. Specific patterns like cryptocurrency addresses, IP addresses, and other unique identifiers are detected and masked out, taking out this potentially harmful information that would cause overfitting while preserving the meaningful content needed for classification.
Solution Approach 2:
The patent converts the potentially harmful effect of identifiers (which cause overfitting) into a benefit by systematically masking them. The masking process transforms documents that would otherwise be unique and cause overfitting into generalized examples that improve model generalization, while the masked placeholders still maintain document structure.
Data Source
AI summary
The present disclosure relates to a method of collecting, by a terminal, learning data of an artificial intelligence model to perform dark web document classification, the method including: generating a first word set masked with a first identifier based on text included in a first dark web document; generating a second word set masked with a second identifier based on text included in a second dark web document; calculating a similarity by comparing the first word set and the second word set; and, based on the similarity, collecting the second dark web document in a second storage unit when the similarity does not exceed a specific value, and discarding the second dark web document when the similarity exceeds the specific value.