Automated Dataset Transformation Script Generation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Big data analysis systems face inefficiencies due to poor quality raw data from disparate sources, which often contains missing fields, anomalies, and formatting issues, limiting the amount of data that can be processed and affecting the quality of analysis results.

Innovation Solution

A system that determines transformation operations to preprocess datasets by generating transformation scripts, identifying and applying specific transformations to improve data quality, allowing users to interactively analyze and refine these operations to enhance data processing efficiency for big data analysis systems.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Manufacturing precision

If transformation scripts are manually developed by experts and developers, then data quality can be improved, but the process becomes tedious and time-consuming

Engineering Contradiction:

Improvedata qualityVSAvoidtime to develop transformation scripts

Core Design Contradiction:

Manufacturing precisionVSLoss of time

Solution Approach 1:

The system automatically generates transformation scripts by analyzing raw data characteristics and determining appropriate transformations needed, eliminating the need for manual script development while maintaining data quality improvements

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent replaces the manual mechanical process of expert analysis and script writing with an automated computational system that performs data analysis and generates transformation scripts algorithmically

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Quantity of substance

If transformation operations are applied to preprocess raw data, then the amount of processable data increases, but the complexity of the preprocessing system increases

Engineering Contradiction:

Improveamount of processable dataVSAvoidpreprocessing system complexity

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

The system segments the preprocessing task into distinct transformation operations that can be independently selected and applied, making the complex preprocessing process more manageable and systematic

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent creates a universal transformation script generation system that can handle multiple types of data quality issues (missing fields, anomalies, formatting problems) through a single automated platform

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS10650020B1Analyzing transformations for preprocessing datasets

Publication Date: 2020.05.12 ALTERYX INC

AI summary

A system analyzes transformations for processing datasets. The transformations may be used for build a transformation script for preprocessing data for analysis by big data analysis systems. The system receives a new transformation for analysis. The system determines a measure of an impact of the new transformation operation on a dataset. The system determines statistical information describing rows of the transformed dataset that are impacted by the new transformation. The system receives request to add the new transformation to the transformation script responsive to presenting the statistical information.