Duplicate Image Detection Using Perceptual Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for detecting duplicate images are computationally expensive and inefficient, particularly when dealing with minor adjustments such as resizing, contrast changes, or cropping, which can lead to increased processing time and resource usage, and can negatively impact machine learning model accuracy by including duplicate images in training and testing datasets.
Innovation Solution
A network-based system and method that uses cryptographic hash functions and advanced image analysis techniques, including perceptual hashing and dimension reduction, to efficiently identify and classify duplicate and potential duplicate images by generating hashes and comparing them against a database, with the option to perform similarity analysis using techniques like Locality-Sensitive Hashing and Twin Neural Networks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional pixel-by-pixel comparison methods are used to detect duplicate images, then measurement precision is improved, but computational cost and processing time increase significantly
Solution Approach 1:
The patent segments the image comparison task into multiple stages: first computing perceptual hashes (pHash) for quick comparison, then using more sophisticated methods like dHash or sHash only when needed. This hierarchical segmentation allows the system to achieve high precision while maintaining computational efficiency by applying heavy processing only to potentially duplicate images.
Solution Approach 2:
The patent introduces perceptual hashing as an intermediary representation between the original image and the final comparison result. By converting images into hash values that capture perceptual similarity, the system can perform rapid preliminary filtering before applying more computationally intensive comparison methods, thus resolving the contradiction between speed and precision.
2Measurement precision
If conventional pixel-by-pixel comparison methods are used to detect duplicate images, then measurement precision is improved, but resource consumption increases significantly
Solution Approach 1:
The patent segments the image comparison task into multiple stages: first computing perceptual hashes (pHash) for quick comparison, then using more sophisticated methods like dHash or sHash only when needed. This hierarchical segmentation allows the system to achieve high precision while maintaining computational efficiency by applying heavy processing only to potentially duplicate images.
Solution Approach 2:
The patent applies partial action by using simplified hash comparison methods for the majority of image pairs and reserving more resource-intensive methods only for cases where the simplified method indicates potential duplication. This selective application of processing intensity significantly reduces overall resource consumption while maintaining high detection accuracy.
3Quantity of substance
If duplicate images are not detected, then storage capacity is fully utilized, but system reliability deteriorates due to processing inefficiency and machine learning model accuracy problems
Solution Approach 1:
The patent performs preliminary duplicate detection using perceptual hashing before images are stored or used in machine learning workflows. By identifying and removing duplicates in advance, the system ensures that only unique, high-quality images are stored and used for training, thereby maintaining both storage efficiency and model accuracy without compromising reliability.
4Loss of time
If duplicate images are not detected, then processing time is reduced, but productivity decreases due to labor-intensive manual evaluation
Solution Approach 1:
The patent implements automated duplicate detection using perceptual hashing and similarity comparison algorithms that operate without human intervention. The system automatically identifies, flags, and removes duplicate images from datasets, eliminating the need for labor-intensive manual evaluation and significantly improving overall processing productivity while maintaining high accuracy.
Data Source
AI summary
A computer system is provided and is programmed to: (1) receive a document; (2) execute a hash function to generate a hash of the document; (3) compare the hash of the document to the plurality of hashes for the plurality of documents; (4) determine if an exact match exists between the hash of the document and the plurality of hashes for the plurality of documents; (5) if an exact match exists, indicate that the received document is a duplicate; and (6) if no exact match exists, the at least one processor is programmed to: (a) perform similarity analysis on the document to compare the document to the plurality of stored documents; (b) determine a similarity measure for the document based on the comparison; (c) compare the similarity measure for the document to a threshold; and (d) indicate that the received document is a potential duplicate based upon the comparison.