Junk Image Detection via URL Pattern Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current multimedia backend systems face inefficiencies due to the need to download and process millions of images daily, where many are 'junk' images with dimensions smaller than a predetermined pixel count, leading to wasted bandwidth and resource utilization.
Innovation Solution
A junk image detection algorithm that utilizes image location information, URL pattern features, webpage content features, click features, and site modeling within a machine learning framework to predict the probability of an image being unwanted, allowing for pre-filtering of image URLs before download, thereby improving indexer bandwidth and publish rate.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the crawler downloads and processes all images to build the index, then the index completeness is improved, but the bandwidth usage and processing time increase significantly
Solution Approach 1:
The patent applies preliminary action by analyzing image metadata (dimensions, file type, source website characteristics) before the crawler downloads the actual image content. This pre-analysis allows the system to identify and filter out junk images (small, low-quality images) before consuming bandwidth to download them, thus maintaining index completeness for valuable images while reducing overall bandwidth usage
Solution Approach 2:
The patent extracts and analyzes specific features from image metadata and source website information without downloading the full image content. By separating the analysis of identifying characteristics from the actual image download process, the system can filter out unwanted images before they consume crawler bandwidth
2Quantity of substance
If the crawler downloads and processes all images, then the index completeness is improved, but the processing time increases
Solution Approach 1:
The system performs preliminary filtering based on image metadata and source characteristics before full processing. By evaluating dimensions, file types, and website reputation metrics in advance, the crawler can skip downloading and processing known junk images, significantly reducing total processing time while maintaining completeness of the valuable image index
Solution Approach 2:
The patent introduces an intermediary filtering layer that sits between the crawler and the full image processing pipeline. This intermediary analyzes lightweight metadata and source features to make go/no-go decisions, acting as a gatekeeper that prevents unnecessary processing of junk images while allowing valuable images to proceed to full indexing
3Productivity
If a simple dimension-based filter is used, then the processing speed is improved, but the accuracy of junk image detection deteriorates
Solution Approach 1:
The patent applies local quality by using different filtering criteria for different contexts. Instead of a single global threshold, the system adjusts filtering parameters based on source website characteristics, image file types, and metadata patterns. This allows the system to maintain high processing speed while improving detection accuracy by adapting to local variations in image quality and source reliability
Solution Approach 2:
The patent combines multiple filtering criteria (image dimensions, file type, source website reputation, URL patterns, metadata characteristics) into a composite filtering approach. Rather than relying on a single dimension-based threshold, the system integrates multiple features to make more accurate junk image detection decisions, achieving both speed and precision through the composite evaluation framework
Data Source
AI summary
Architecture that includes a junk (unwanted) image detection algorithm which performs junk image detection of unwanted images before the images are actually downloaded for indexing. Features are employed related to image location information and host websites, such as image path descriptor (e.g., URL-uniform resource locator) pattern features, webpage content features, click features, and image aggregated information in a machine learning based framework to predict the probability that an image is unwanted (or wanted) before the images are downloaded. The framework is then applied to build a statistical model and predict junk scores. By removing image URLs marked as “junk” from the work list of an automated indexer (e.g., crawler), the indexer bandwidth is significantly improved with a corresponding improvement in the publish rate.