Interactive Data Column Splitting via Dynamic Program Synthesis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data cleaning and wrangling processes are time-consuming and require programming skills, especially when dealing with data in varying formats, as traditional tools struggle to handle arbitrary and varying formats, and conventional programming-by-example techniques require extensive user input and may produce over-specific programs.
Innovation Solution
An interactive data splitting system that synthesizes a program to split an input column into multiple output columns based on input-only examples, allowing user input to guide the synthesis and refinement of the program, and executes it to produce a formatted data set, using a domain-specific language and interactive techniques to handle ambiguities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional file parsers are used to split data into columns, then standardized format data can be processed, but data with arbitrary and varying formats cannot be handled
Solution Approach 1:
The system dynamically adapts the splitting program based on input data characteristics. Instead of using a fixed parser, the system generates and refines splitting programs that adjust to the specific format patterns detected in the input data, allowing reliable handling of varying formats through dynamic program synthesis and iteration.
Solution Approach 2:
The system changes the parameters of the splitting operation by generating multiple candidate programs with different splitting strategies. By varying parameters such as delimiter selection, split positions, and program logic, the system can adapt to different data formats while maintaining accuracy through iterative refinement based on user feedback.
2Ease of operation
If programming-by-example techniques are used to automate data cleaning, then programming skills are reduced, but extensive user input is required
Solution Approach 1:
The system performs preliminary analysis of the input data to automatically detect format patterns and generate initial splitting programs without requiring extensive user examples. By pre-processing the data to identify delimiters, field structures, and formatting conventions, the system reduces the burden on users while maintaining ease of operation.
Solution Approach 2:
The system implements iterative feedback loops where user corrections and refinements to generated programs are automatically incorporated. This feedback mechanism allows the system to learn from user interactions and improve subsequent program generation, reducing the overall time required for user input while maintaining accessibility.
3Measurement precision
If conventional PBE techniques use multiple input-output examples per field, then extraction accuracy improves, but manual effort increases proportionally
Solution Approach 1:
The system uses a minimal number of input examples (as few as one per field) to generate initial splitting programs, then applies iterative refinement and validation processes to achieve high extraction accuracy. This partial action approach avoids the need for extensive examples while maintaining precision through automated program improvement.
Solution Approach 2:
The system maintains continuous improvement of the splitting program through automated iterative refinement processes. Instead of requiring discrete batches of examples, the system continuously optimizes the program based on data patterns and user feedback, improving both accuracy and productivity through uninterrupted automated processing.
4Ease of operation
If users provide examples from the top rows of data, then the process is simplified, but the learned program becomes over-specific and fails on later rows
Solution Approach 1:
The system generates splitting programs that are designed to be universal rather than specific to particular data rows. By analyzing patterns across the entire dataset and using iterative refinement with samples from various positions, the system creates programs that generalize well to different data formats and positions, avoiding over-specificity while maintaining ease of operation.
Solution Approach 2:
The system performs preliminary analysis of the entire dataset to identify consistent formatting patterns before generating the splitting program. This preliminary action ensures that the program learns from diverse data positions rather than just top rows, improving generalization capability while keeping user effort minimal through automated pattern detection.
Data Source
AI summary
Various technologies described herein pertain to interactive data splitting. A program for splitting an input column of an input data set into multiple output columns can be synthesized based on input-only examples. The program can further be generated based on various user input; thus, the user input can guide the synthesis of the program. Moreover, the program can be executed on the input data set to split the input column of the input data set into the multiple output columns.