Nonlinear Feature Detection for Lower-Burden Machine Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing amount of data input into machine learning models leads to increased resource and time consumption, with duplicate and highly correlated data not contributing significantly to model learning, resulting in wasted resources and time.
Innovation Solution
A method involving data set reduction by identifying and removing duplicate and highly correlated data using Singular Value Decomposition (SVD) and Pearson's co-variance (PCV) after linearizing the data set to detect non-linear relationships, followed by Delaunay triangulation for further refinement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the amount of data inputted into the machine learning model is increased to improve model robustness, then the robustness of the machine learning model is improved, but the resources and time required to create and enhance the model increase significantly
Solution Approach 1:
The patent extracts and removes duplicate data and highly correlated data from the input dataset before feeding it to the machine learning model. This extraction process identifies redundant information (duplicate columns) and strongly related information (highly correlated columns with correlation coefficient > threshold), then removes them to reduce processing time while maintaining model robustness.
Solution Approach 2:
The patent changes the parameter of data quantity by applying data reduction techniques. It transforms the original large dataset into a reduced dataset by removing redundant and highly correlated columns, thereby changing the effective data quantity parameter to optimize the balance between model robustness and processing time.
2Measurement precision
If the granularity of data review is increased to improve model accuracy, then the accuracy of the machine learning model is improved, but the resources and time required to process the data increase
Solution Approach 1:
The patent extracts only the essential and non-redundant features from the dataset by removing duplicate and highly correlated columns. This extraction ensures that the model processes only necessary data with sufficient granularity for accurate learning, avoiding wasteful computation on redundant information.
Solution Approach 2:
The patent applies partial action by selectively processing only the most relevant data columns rather than reviewing all data with maximum granularity. It uses correlation thresholds to determine which columns require detailed review, applying computational resources partially to the most important features.
3Quantity of substance
If duplicate and highly correlated data are retained in the dataset, then the completeness of the input data is maintained, but the machine learning model does not learn significantly from this data resulting in wasted resources
Solution Approach 1:
The patent extracts and removes duplicate data (columns with duplicate names) and highly correlated data (columns with correlation coefficient greater than a predetermined threshold) from the dataset. This extraction maintains data completeness by preserving all unique and informative features while eliminating redundant information that hinders learning efficiency.
Solution Approach 2:
The patent discards redundant data (duplicates and highly correlated columns) that do not contribute to model learning, while recovering and preserving the essential unique features. This selective discarding and recovering process optimizes the dataset for efficient model training without losing important information.
Data Source
AI summary
Apparatus and methods for detecting non-linear data dependencies in machine learning are provided. The methods may include receiving a data set comprising a plurality of data features, the data set for processing by a machine learning model operating on a machine learning system. The methods may also include plotting the data set in multi-dimensional space and linearizing the plotted, multi-dimensional space by feeding the plotted data to an algorithm programmed to execute the Delaunay triangulation method. The methods may further include receiving from the algorithm the data set linearized in a faceted space and detecting linear correlations in the linearized data set. The methods may additional include identifying a first data feature that has a correlation coefficient with a second data feature that is greater than an accuracy metric and removing the first data feature from the data set to create a reduced data set.