Image Dataset Cleaning with IoU Filtering for Redundant Frame Removal
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for managing large volumes of image data in automation technology, particularly in automatic unpacking machines and autonomous vehicles, struggle with semantic redundancy, leading to inefficient data processing and loss of relevant information.
Innovation Solution
A method utilizing an intersection-over-union (IoU) filter to identify and remove redundant images by comparing keyframes with a predetermined threshold, focusing on foreground changes and maintaining dataset quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If semantic redundancy filtering is applied to reduce data volume, then processing efficiency improves, but risk of losing relevant information increases
Solution Approach 1:
The patent replaces traditional mechanical redundancy filtering methods with a deep learning-based semantic analysis system. The neural network model analyzes the semantic content of images to identify and remove redundant data, substituting mechanical filtering with intelligent semantic understanding to maintain information quality while improving processing efficiency
Solution Approach 2:
The patent changes the parameter of redundancy detection from simple visual similarity to semantic content similarity. By using deep learning models to analyze semantic parameters rather than basic image parameters, the system can more accurately distinguish between redundant and valuable data, reducing information loss while maintaining high processing efficiency
2Measurement precision
If deep learning models are trained on large volumes of image data, then model accuracy improves, but training time and computational resources increase
Solution Approach 1:
The patent extracts and removes redundant images from the training dataset using semantic redundancy filtering. By taking out duplicate and highly similar images that provide minimal additional learning value, the system reduces the total training data volume while preserving the most informative samples, thereby decreasing training time without significantly compromising model accuracy
Solution Approach 2:
The patent performs preliminary redundancy filtering on the image dataset before training the deep learning model. This preliminary action of removing redundant data beforehand reduces the computational burden during training, allowing the model to learn from high-quality, non-redundant samples more efficiently, thus reducing training time while maintaining accuracy
Data Source
AI summary
A method and apparatus (100) for cleaning an image data set used for training and/or validating and/or testing a machine learning model is proposed, the method comprising the steps of: - providing (S1) the image data set, comprising a plurality of images; - comparing (S2) a, in particular predetermined, comparison image from the plurality of images with at least a subset of the remaining images of the plurality of images by applying an intersection-over-union filter; - based on the comparison, identifying (S3) at least one image redundant with respect to the comparison image in at least the subset of the remaining images of the plurality of images; and - cleaning (S4) the image data set by removing the at least one redundant image from the plurality of images.