Semi-Synthetic Data Generation for Reliable Machine Learning Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning systems face challenges in fields with insufficient training data, where accuracy is affected by data quantity, quality, and processing speed, particularly due to overfitting, underfitting, and the presence of impure data, limiting their effectiveness and reliability.
Innovation Solution
A semi-synthetic data generation apparatus and method that includes data preprocessing, learning model training, validation, and a semi-synthetic data generation unit to create reliable data by selecting parameters within defined boundaries, filtering out impure data, and optimizing hyperparameters using cross-validation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data augmentation methods are used to generate sufficient training data, then the amount of training data is increased, but the processing time and computational resources increase significantly
Solution Approach 1:
The system performs preliminary actions by pre-processing the initial dataset to extract feature spaces and decision boundaries before actual data generation. This preparation work is done once upfront, allowing rapid generation of semi-synthetic data without repeating complex computations during the data augmentation process.
Solution Approach 2:
The invention changes parameters by transforming the problem from generating complete data samples to generating only new parameter combinations within identified feature spaces. This parameter-based approach generates data by combining existing features with new parameter values, significantly reducing computational complexity while maintaining data quality.
2Quantity of substance
If traditional data augmentation methods are applied to various fields, then data quantity is increased, but the reliability and validity of generated data remain insufficient
Solution Approach 1:
The system implements feedback mechanisms by using trained machine learning models to validate and filter generated semi-synthetic data. The models assess whether generated data points conform to learned patterns and decision boundaries, providing feedback that ensures data reliability while maintaining augmentation effectiveness.
Solution Approach 2:
The invention introduces an intermediary approach by using feature spaces and decision boundaries as mediators between raw data and generated data. These intermediaries represent the underlying structure and relationships in the data, ensuring that generated samples maintain consistency with the original data distribution and patterns.
3Productivity
If machine learning systems are trained with insufficient data, then training speed is faster, but accuracy and reliability of the system decrease
Solution Approach 1:
The system creates high-quality copies of existing data through semi-synthetic generation rather than simply duplicating samples. By copying the structural patterns, relationships, and feature distributions from the initial dataset and combining them with newly generated parameters, the system produces diverse yet reliable training data that improves model accuracy without requiring proportional increases in training time.
Data Source
AI summary
A data generation apparatus includes a data input unit that inputs an initial dataset, a data preprocessing unit that normalizes the initial dataset and splits the normalized initial dataset into a initial training dataset and a initial test dataset, a learning model generation unit that trains a first machine learning model using the initial training dataset and optimizes a hyperparameter of the first machine learning model using a predetermined number of cross-validations, thereby generating a first learning model, a validation unit that validates the generated first learning model using the initial test dataset, a semi-synthetic data generation unit that selects and generates a new parameter within a boundary space defined by the initial dataset and performs data prediction using the first learning model based on the new parameter to generate a semi-synthetized dataset, and a database that stores the initial dataset and the semi-synthetized dataset.