Automated Data Preparation for Machine Learning Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning systems require manual operations for data transformation and processing, which is time-consuming and inefficient, especially when preparing training, validation, and test data for consistent use with the initial training data.
Innovation Solution
A method for automatically preparing data for machine learning systems by identifying column labels, determining root categories, performing data transformations based on a transformation tree, and recording metadata for consistent processing of additional data sets, including options for infill, feature importance, and dimensionality reduction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If manual data transformation operations are performed, then data can be processed according to specific requirements, but the process is time-consuming and inefficient
Solution Approach 1:
The system automatically determines transformation parameters and executes data transformations without requiring manual intervention. The automated pipeline analyzes training data properties, selects appropriate transformations, and applies them consistently to both training and additional data sets, eliminating time-consuming manual operations while maintaining transformation precision
Solution Approach 2:
The system dynamically adjusts transformation parameters based on the statistical properties of the input data. By automatically detecting data characteristics such as distribution, missing values, and feature types, the system selects and applies optimal transformation parameters to normalize and standardize data consistently across different data sets
2Manufacturing precision
If manual data processing is used, then specific transformation requirements can be met, but productivity is reduced
Solution Approach 1:
The automated system performs data transformation tasks independently by analyzing training data properties and executing transformations without human intervention. This self-service capability maintains data consistency through standardized transformation pipelines while significantly improving productivity by eliminating manual processing steps
Solution Approach 2:
The system performs preliminary analysis of training data to automatically determine transformation parameters before processing additional data sets. By pre-establishing transformation rules and parameters based on training data properties, the system enables rapid, consistent processing of subsequent data without requiring manual reconfiguration
3Productivity
If automated data transformation is implemented, then processing efficiency improves, but data leakage may occur between training, validation, and test data
Solution Approach 1:
The system segments the data processing pipeline into distinct phases: training data analysis, parameter determination, and transformation application. By separating these phases and using the training data only for parameter determination (not transformation parameter tuning), the system maintains processing efficiency while preventing data leakage between training, validation, and test sets
Solution Approach 2:
The system performs preliminary transformation parameter determination using only training data properties, then applies these pre-determined parameters consistently to validation and test data. This preliminary action approach ensures that no information from validation or test data influences the transformation parameters, preventing data leakage while maintaining automated processing
4Manufacturing precision
If multiple data transformations are applied, then data quality improves, but the complexity of the transformation process increases
Solution Approach 1:
The transformation process is segmented into discrete, manageable steps based on data properties and root categories. Each transformation operation is independently defined and can be applied sequentially, making the overall complex process transparent and controllable while improving data quality through multiple specialized transformations
Solution Approach 2:
The system automatically selects transformation parameters based on detected data properties, reducing the need for manual configuration of complex transformation pipelines. By dynamically adjusting parameters according to data characteristics, the system maintains high data quality while simplifying the user interface and reducing operational complexity
Data Source
AI summary
A technique for automated preparation of tabular data for machine learning, including options for machine learning derived infill, feature importance evaluations, and/or dimensionality reduction. Validation data sets may be consistently prepared to training data sets based on properties of the training data saved in a metadata database. Additional data sets may be consistently prepared to training data sets based on properties of the training data saved in a returned metadata database such as for use in generating predictions from the trained ML system. Returned data sets may be prepared for oversampling of labels with lower frequency occurrence. Columns of a training data set are evaluated for appropriate categories of transformations, with the composition of transformation function applications designated by a defined tree of transformation category assignments to transformation primitives. Composition of transformation trees and their associated transformation functions may optionally be custom defined by a user.