Automated Feature Extraction from Secondary Data for ML
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The manual design and construction of training datasets for machine learning processes are time-consuming, expensive, and error-prone, especially when dealing with large datasets from diverse sources, as data scientists struggle to identify relevant features for accurate prediction and classification.
Innovation Solution
A computer-implemented method that automatically generates a set of pivotal classification features by linking relevant secondary data fields to a primary training dataset, selecting statistically significant features, and creating an enhanced training dataset for use in automated machine learning processes, thereby improving classification accuracy and reducing computational resource requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual design and construction of training datasets is performed, then classification accuracy can be improved through careful feature selection, but time consumption and cost increase significantly
Solution Approach 1:
The system performs automatic feature extraction and dataset construction without requiring manual intervention. The machine learning model autonomously identifies relevant features from secondary datasets and constructs the enhanced training dataset, eliminating the need for data scientists to manually design and construct training datasets while maintaining classification accuracy
Solution Approach 2:
The manual mechanical process of feature selection and dataset construction by data scientists is replaced with an automated computational system. The system uses correlation analysis and automated feature extraction algorithms to substitute the manual workflow, significantly reducing time consumption while preserving the quality of feature selection
2Measurement precision
If more data fields are included in the training dataset, then classification accuracy may improve, but computational resource requirements increase
Solution Approach 1:
The system extracts only the most relevant features from secondary datasets based on correlation analysis with the target variable. By taking out and selecting only the pivotal features that contribute to classification accuracy, the system avoids including unnecessary data fields that would increase computational resource requirements while maintaining or improving classification performance
Solution Approach 2:
The system applies different processing quality to different features based on their relevance. High-correlation features are extracted and included in the enhanced training dataset, while low-correlation features are discarded. This local quality approach ensures that computational resources are focused on processing only the most valuable features, optimizing the balance between accuracy and resource usage
3Productivity
If automated feature extraction from secondary datasets is implemented, then time consumption is reduced, but system complexity increases
Solution Approach 1:
The system performs preliminary correlation analysis between available secondary dataset fields and the target variable before feature extraction. This preliminary action identifies which secondary datasets are relevant and which fields within them are most useful, simplifying the subsequent automated feature extraction process by focusing only on promising data sources rather than processing all possible fields
Data Source
AI summary
A classifier is computed as follows. For a first set of values of primary field(s) of primary data instances of a labeled primary training dataset, a second set(s) of secondary fields of unclassified second data instances of secondary dataset(s) is identified. First set of values are matched to corresponding values in respective secondary field(s), and linked to other secondary fields of respective secondary data instance(s) of the respective matched secondary field. A set of classification features is generated. Each including: (i) condition(s), and (ii) a value selected from the linked other secondary fields of the respective secondary data instance(s) of the respective matched secondary field(s). The respective classification feature outputs a binary value computed by the condition(s) that compares between the value selected from the other linked secondary fields and a new received data instance. A classifier is computed according to a selected subset of the classification features.