Feature Processing Recipes for Machine Learning Services
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Business organizations face barriers in utilizing machine learning due to the requirement of statistical and AI expertise, computational intensity, and challenges in handling large datasets, especially for infrequent but significant events like financial fraud detection, where traditional machine learning environments lack optimal data transformation mechanisms.
Innovation Solution
A customizable machine learning service (MLS) with programmatic interfaces guides non-expert users through machine learning best practices, allowing default settings and enabling expert customization, featuring a recipe language for data transformations, asynchronous job scheduling, and secure, scalable processing plans across a provider network with distributed computing resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If machine learning models are built using traditional methods, then predictive accuracy can be achieved, but the requirement for statistical and AI expertise creates a high barrier for business analysts and subject matter experts
Solution Approach 1:
The patent introduces an automated machine learning service that acts as an intermediary between business analysts and complex ML algorithms. The service includes automated data preprocessing, feature engineering, model selection, and hyperparameter tuning components that translate business requirements into ML models without requiring users to have statistical or AI expertise. This mediator handles the complexity internally while providing simple interfaces to users.
Solution Approach 2:
The system enables self-service machine learning by automatically performing tasks that traditionally required expert intervention. The automated ML service autonomously conducts data exploration, selects appropriate algorithms, optimizes parameters, and generates models based on user-provided business objectives and data, allowing non-experts to independently build and deploy ML solutions.
2Measurement precision
If machine learning techniques are applied to large datasets, then accurate predictions can be made, but the computational intensity requires significant computing power that may not be available to business organizations
Solution Approach 1:
The system implements automated data sampling and feature selection techniques that process only the most relevant portions of large datasets. By identifying and focusing on key features and representative data subsets, the system achieves accurate predictions with reduced computational requirements, avoiding the need to process entire massive datasets in detail.
Solution Approach 2:
The automated ML service dynamically adjusts computational parameters such as sample size, feature dimensionality, model complexity, and processing depth based on data characteristics and prediction requirements. This adaptive parameter adjustment allows the system to optimize the balance between prediction accuracy and computational resource consumption for different business scenarios.
3Reliability
If transformations are applied to input data variables in traditional machine learning environments, then data quality improves, but the mechanisms available are less than optimal and require tedious manual work
Solution Approach 1:
The system performs preliminary automated data transformations and preprocessing steps before model training. This includes automatic data cleaning, normalization, feature engineering, and validation procedures that are executed upfront based on data exploration results, eliminating the need for iterative manual transformation adjustments and improving both data quality and processing efficiency.
Data Source
AI summary
A first representation of a feature processing recipe is received at a machine learning service. The recipe includes a section in which groups of variables on which common transformations are to be applied are defined, and a section in which a set of transformation operations are specified. The first representation of the recipe is validated based at least in part on a library of function definitions supported by the service, and an executable version of the recipe is generated. In response to a determination that the recipe is to be executed on a particular data set, a set of provider network resources is used to implement a transformation operation indicated in the recipe.