Near-Duplicate Image Filtering for Unbiased ML Training Data

The patent extracts and removes near-duplicate images from the training data using feature vector clustering and cost matrix analysis. By identifying images with high correlation coefficients and excluding them, the system maintains adequate training data quantity while eliminating the harmful effect of near-duplicates that cause bias and overestimation.

The patent introduces feature vectors as an intermediary representation between images and the training process. By converting images to feature vectors and using these vectors for similarity comparison and clustering, the system efficiently identifies near-duplicates without directly comparing all image pairs, thus maintaining data quality while preserving training effectiveness.

The patent segments the image comparison process into multiple stages: first clustering images based on feature vectors, then generating cost matrices only for clusters with potential near-duplicates, and finally applying Hungarian algorithm only where needed. This segmentation reduces the computational complexity compared to a brute-force approach while maintaining high data quality.

The patent applies partial action by using threshold-based filtering to identify only the most suspicious near-duplicate pairs for detailed analysis. By setting a correlation coefficient threshold and only applying the Hungarian algorithm to pairs exceeding this threshold, the system achieves high detection accuracy while reducing unnecessary computational overhead.