Automated Data Preparation System for Model Development

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

The data preparation phase for model development is time-consuming and challenging due to the difficulty in understanding and interpreting input data's impact on target behavior, requiring extensive analysis and effort to profile variables effectively.

Innovation Solution

A method and system for automated data preparation that creates new variables by extracting relevant data from sources, processing it to generate input variables, displaying processed data interactively, modifying parameters, executing code to capture processing steps, and storing the code for later use, enabling efficient and informative data handling and visualization.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If automated data extraction and transformation is implemented, then productivity and efficiency are improved, but device complexity increases

Engineering Contradiction:

Improvedata preparation efficiencyVSAvoidsystem complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent introduces an automated data preparation system that acts as an intermediary between raw data sources and model development processes. This system includes components for automatic variable generation, data transformation, and code execution that mediate the complex data preparation tasks, thereby improving productivity while managing system complexity through structured automation.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system implements self-service capabilities through automated variable generation from natural language descriptions, automatic code execution, and self-documented processing steps. The system can autonomously perform data extraction, transformation, and variable creation without requiring manual intervention for each step, thus enhancing productivity while the automation framework manages the underlying complexity.

Inventive Principle:

Principle #25Self-service

2Measurement precision

If extensive data analysis is performed to understand variable impact, then measurement precision is improved, but loss of time increases

Engineering Contradiction:

Improvevariable impact understandingVSAvoiddata preparation time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system performs preliminary actions by automatically generating variables and preparing data structures before model development begins. It pre-processes data, creates transformation codes, and organizes variables in advance, which reduces the time required during actual model development while maintaining comprehensive analysis capabilities for understanding variable impacts on target behavior.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent replaces manual mechanical analysis processes with automated computational systems. Instead of manually analyzing each variable's impact through extensive human effort, the system uses automated code generation, execution, and transformation processes that can rapidly assess variable impacts, thereby maintaining measurement precision while dramatically reducing the time investment required.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Reliability

If comprehensive variable profiling is conducted, then reliability is improved, but loss of time increases

Engineering Contradiction:

Improvemodel development reliabilityVSAvoidvariable analysis time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system enables continuous useful action by implementing automated, ongoing data preparation processes that can run continuously without interruption. The automated variable generation, data transformation, and profiling operations can execute continuously as data becomes available, ensuring comprehensive and reliable variable analysis while reducing the time researchers need to invest through non-stop automated processing.

Inventive Principle:

Principle #20Continuity of useful action

Solution Approach 2:

The patent substitutes manual variable profiling mechanisms with automated computational processes. The system automatically generates, transforms, and analyzes variables using programmed algorithms that can comprehensively profile all variables without the time constraints of manual analysis, thereby maintaining high reliability in model development while dramatically reducing the time required for comprehensive variable assessment.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS10289704B1Method and system for implementing categorically organized relationship effects

Publication Date: 2019.05.14 JPMORGAN CHASE BANK NA

AI summary

According to an embodiment of the present invention, a computer implemented method and system for developing variables for model generation comprises: initiating, via an input to a computer, creation of a new variable for a dataset for model generation; extracting, by the computer, data relevant to the variable from one or more data sources; processing, by the computer, the extracted data to automatically generate an input variable; displaying, via a user interactive interface, the processed data relative to a target variable; modifying, via the user interactive interface, one or more parameters that define the input variable; executing, by the computer, the input variable by extracting code that captures the processing step to generate the input variable; and storing, in a database, the extracted code for the input variable.