Concurrent Quantile Binning for Machine Learning Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Business organizations face barriers in utilizing machine learning due to the requirement of statistical and AI expertise, computational intensity, and challenges in handling large datasets, especially for infrequent event predictions, such as financial fraud detection, where traditional mechanisms for data transformation are inefficient and error-prone.
Innovation Solution
A customizable machine learning service (MLS) with programmatic interfaces that guides non-expert users through machine learning best practices, allowing default settings derived from accumulated experience, and enabling expert customization, along with automated data processing and model training, using a provider network with distributed computing and security features to handle large datasets efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional data transformation mechanisms are used, then transformations can be applied to data variables, but the process is tedious and error-prone when applied one by one to many different variables
Solution Approach 1:
The patent combines multiple sequential data transformation operations into a single unified transformation step. The system applies transformations to groups of variables simultaneously rather than one by one, merging the transformation process into an atomic operation that reduces both time consumption and error probability while maintaining the same functional outcome.
2Measurement precision
If large datasets are analyzed for infrequent event predictions, then prediction accuracy improves, but computational requirements and data handling complexity increase significantly
Solution Approach 1:
The patent segments large datasets into manageable chunks or partitions that can be processed independently. By dividing the data processing task into smaller segments, the system can handle large datasets for infrequent event prediction without overwhelming computational resources, while still maintaining the ability to analyze all necessary data for accurate predictions.
3Measurement precision
If machine learning models are customized for specific problems, then model accuracy improves, but the requirement for statistical and AI expertise increases
Solution Approach 1:
The patent implements automated model selection and configuration mechanisms that allow the system to automatically determine appropriate machine learning models and their parameters based on the problem characteristics. This self-service approach enables non-experts to achieve accurate customized models without requiring deep statistical or AI knowledge, as the system autonomously performs the customization process.
Data Source
AI summary
Variables of observation records to be used to generate a machine learning model are identified as candidates for quantile binning transformations. In accordance with a particular concurrent binning plan generated for a particular variable, a plurality of quantile binning transformations are applied to the particular variable, including a first transformation with a first bin count and a second transformation with a different bin count. The first and second transformations result in the inclusion of respective parameters or weights for binned features in a parameter vector of the model. In a post-training phase run of the model, at least one parameter corresponding to a binned feature is used to generate a prediction.