Dataset Subset Generation for SAR Image Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training machine learning algorithms on sensor data, such as SAR satellite images, face challenges in providing consistent and balanced training and validation data sets, particularly when dealing with large and dynamic datasets.
Innovation Solution
A computer-implemented method for generating subsets of a data set by receiving a data set, applying augmentations to the data elements, and training a machine learning module to determine feature representations. The method splits the data set into subsets to minimize dissimilarity between their distributions, allowing for efficient training and validation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual curation is used to select training and validation data sets, then the data sets can be carefully selected, but the process is time-consuming and creates a prohibitive drain on human labour
Solution Approach 1:
The patent replaces manual curation processes with automated machine learning-based systems. The machine learning module automatically selects and balances training and validation data sets by analyzing feature representations and computing distances between data elements, eliminating the need for human curators to manually review and select data while maintaining high selection quality.
Solution Approach 2:
The system performs self-service by automatically evaluating and selecting data sets based on their intrinsic properties. The machine learning module computes feature representations and distances to automatically determine optimal data splits, allowing the system to self-optimize without human intervention and significantly reducing the time and labor required for data set curation.
2Reliability
If manual sorting of data elements is used to create subsets, then human expertise can be applied, but the process is slow and introduces human bias
Solution Approach 1:
The patent replaces manual sorting with automated computational methods. The machine learning module calculates feature representations and distances between data elements, then algorithmically partitions the data into balanced subsets. This automated approach eliminates human bias in selection while dramatically increasing the speed of subset generation compared to manual sorting processes.
Solution Approach 2:
The system transforms the manual sorting process into an automated parameter-driven process. By computing quantitative distances and feature representations, the system objectively determines optimal subset assignments based on mathematical criteria rather than human judgment, ensuring consistency and eliminating bias while operating at machine processing speeds.
3Reliability
If the data set is split into subsets for training and validation, then consistent evaluation can be performed, but ensuring balanced and representative subsets is difficult
Solution Approach 1:
The patent replaces complex manual balancing processes with automated machine learning algorithms. The system computes feature representations for all data elements, calculates distances between them, and uses these metrics to algorithmically partition the data into balanced subsets. This automated approach ensures consistent and representative splits while simplifying the overall process compared to manual balancing methods.
Solution Approach 2:
The system incorporates feedback mechanisms to ensure balanced subset creation. By computing distances and feature representations, the system can evaluate and adjust subset compositions to maintain balance and representativeness. This feedback-driven approach ensures that validation subsets are consistently representative of the overall data distribution, improving validation reliability.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
A computer-implemented method of generating subsets of a data set, the method comprising: determining an input data set comprising a plurality of data elements, wherein the data elements comprise one or more features; and splitting the data set into a first subset and a second subset such that a dissimilarity between the distributions of aggregated features included in each of the subsets is minimized.