Automated Data Column Splitting via Predictive Program Synthesis

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data cleaning and wrangling processes are time-consuming and require programming skills, as they rely on manual effort and explicit intent specification, making it difficult to handle data with varying formats and delimiters.

Innovation Solution

Automated data splitting using predictive program synthesis, which receives input-only examples to identify patterns and synthesize programs in a domain-specific language (DSL) for splitting data columns without explicit specification, employing contextual and zero-length delimiters.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If traditional file parsers for CSV files are used to split data, then standardized format data can be processed, but data with arbitrary and varying formats cannot be handled

Engineering Contradiction:

Improveability to handle varying data formatsVSAvoidcomplexity of parsing system

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system dynamically changes parsing parameters (delimiter characters, field separators) based on the detected format of each data entry. Instead of using a fixed parser, the system adapts parameters like the delimiter character and splitting rules to match the specific format characteristics of the input data, enabling handling of arbitrary formats while maintaining a unified parsing framework

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The parsing system transitions from a static, pre-configured approach to a dynamic, adaptive approach. The system detects format patterns in the input data and adjusts its parsing behavior in real-time, making the parser flexible enough to handle varying formats without requiring multiple specialized parsers for different data types

Inventive Principle:

Principle #15Dynamics

2Productivity

If manual data cleaning and wrangling processes are used, then data can be normalized and validated, but the process is time-consuming and requires programming skills

Engineering Contradiction:

Improvedata cleaning speedVSAvoidease of data cleaning operation

Core Design Contradiction:

ProductivityVSEase of operation

Solution Approach 1:

The system performs automatic format detection and parsing without requiring manual specification of parsing rules. The data itself provides the information needed to determine how it should be split, eliminating the need for users to write extraction or transformation scripts while maintaining high productivity in the data cleaning process

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system replaces manual mechanical data cleaning operations with an automated intelligent system. Instead of requiring users to manually specify delimiters and splitting rules, the system uses format detection algorithms to automatically determine the appropriate parsing strategy, making data cleaning accessible to non-programmers

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Extent of automation

If programming-by-example approaches are used for automatic data cleaning, then programs can be generated, but explicit intent specification and manual effort are still required

Engineering Contradiction:

Improveautomation of data splittingVSAvoidtime for providing input-output examples

Core Design Contradiction:

Extent of automationVSLoss of time

Solution Approach 1:

The system extracts format information directly from the data entries themselves rather than requiring users to provide separate input-output examples. By detecting delimiters, field structures, and format patterns directly in the input data, the system eliminates the time-consuming step of manually creating example pairs while achieving the same automation goal

Inventive Principle:

Principle #2Taking out (Extraction)

Data Source

PatentUS20240143562A1Automatic splitting of a column into multiple columns

Publication Date: 2024.05.02 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

Various technologies described herein pertain to automated data splitting using predictive program synthesis. Input-only examples for splitting an input column of an input data set can be received. The input-only examples can include example entries from the input column of the input data set to be split into multiple output columns without specification of how the example entries are to be split into multiple output columns. Further, a program for splitting the input column of the input data set into the multiple output columns can be synthesized based on the input-only examples. The program can be synthesized, given the input-only examples, in a domain-specific language (DSL) for splitting an entry into a tuple of fields utilizing a predictive program synthesis algorithm. Moreover, the program can be executed on the input data set to split the input column of the input data set into the multiple output columns.