Synthetic Training Data Schema Mapping for Diverse ML Records
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training machine learning models is challenging due to the difficulty in converting diverse input data sets to a common structure and format, which affects the number and quality of training samples needed for accurate model performance.
Innovation Solution
A training schema is used to organize a training data set, with a software application transforming data records from varying schemas to a common format through predefined mappings and generating files in specific formats, and synthesizing missing values to enhance data diversity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple different input data sets with different schemas are used to generate training data, then the diversity and quantity of training samples increase, but the difficulty of converting different schemas to a common format increases
Solution Approach 1:
The patent introduces a schema mapping system as an intermediary layer between diverse input data sets and the training data generation process. This mapping system includes configurable mapping rules that automatically transform different source schemas to a target schema, eliminating the need for manual conversion and reducing complexity while maintaining data diversity
Solution Approach 2:
The patent creates a universal target schema that can accommodate multiple different source schemas. The training data generation system is designed to handle various input formats (JSON, XML, CSV, etc.) and convert them all to a common structure, making the system multi-functional and adaptable to diverse data sources
2Manufacturing precision
If data records are transformed from varying schemas to a common training schema, then the quality and consistency of training data improve, but the time and computational resources required for transformation increase
Solution Approach 1:
The patent performs schema mapping and data transformation in advance during the training data generation phase, before the actual machine learning training begins. By pre-processing and organizing data into the target schema beforehand, the system eliminates the need for repeated transformations during model training, saving significant time and computational resources
Solution Approach 2:
The patent creates copies of source data records and transforms them into the target schema format. Rather than modifying original data sources, the system generates transformed copies that conform to the training schema, allowing efficient batch processing and parallel transformation of multiple data sets
3Quantity of substance
If missing fields and values are synthesized to complete data records, then the completeness and informational diversity of the training data set increase, but the complexity of data processing and validation increases
Solution Approach 1:
The patent implements automated field synthesis where the system itself generates missing values using synthesis rules configured within the training data generation process. The system automatically identifies missing fields, applies appropriate synthesis strategies (such as generating placeholder values or deriving values from related fields), and validates the completed records without requiring external intervention
Solution Approach 2:
The patent changes the state of incomplete data records by applying synthesis operations that transform missing or null values into concrete data values. The system dynamically adjusts data parameters by generating synthetic values that match the expected data types and formats, converting incomplete records into complete, training-ready records
Data Source
AI summary
A method includes obtaining first data records associated with a first schema. Each respective data record of the first data records may include corresponding values of a plurality of fields. The method also includes obtaining a mapping representing a transformation of a field of the plurality of fields from the first schema to a training schema, and selecting, based on the mapping and for each respective data record of the first data records, a corresponding value of the field from the first data records. The method further includes generating second data records by transforming, based on the mapping and for each respective data record of the first data records, the corresponding value of the field from the first schema to the training schema. The method additionally includes training a machine learning model based on the second data records.