Transform-by-pattern system for automated data transformation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Non-technical users face challenges in data transformation tasks due to the need for manual input of examples and expertise in writing programs, which burdens them in various applications, especially in transforming data between different formats or languages.
Innovation Solution
A transform-by-pattern (TBP) system that proactively suggests relevant transformation programs based on input/output datasets without requiring users to type in examples, by generating TBP programs offline and making them available online for automatic data transformation, using a combination of source and target patterns and transformation programs to identify applicable programs for related data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If users manually input examples and write programs for data transformation, then transformation accuracy can be achieved, but user burden and operation complexity increase
Solution Approach 1:
The system enables self-service data transformation by automatically analyzing source and target data patterns, generating transformation programs without requiring user input of examples or programming knowledge. The system serves itself by autonomously identifying patterns and creating transformation logic.
Solution Approach 2:
The system introduces an intermediary pattern-matching mechanism that mediates between source data and target data. Instead of direct user-program-data interaction, the system uses learned patterns as intermediaries to automatically bridge the transformation gap.
2Productivity
If users manually identify columns and enter examples for transformation, then specific transformation tasks can be completed, but time consumption and productivity decrease
Solution Approach 1:
The system performs preliminary action by pre-learning data patterns and transformation relationships from historical data pairs. This offline learning phase prepares transformation programs in advance, so when actual transformation is needed, the system can quickly apply pre-learned patterns without time-consuming manual analysis.
Solution Approach 2:
The system maintains continuity of useful action by continuously learning from new data pairs and refining transformation patterns. The learned patterns are continuously applied to transform data, creating an uninterrupted automated transformation process that eliminates manual intervention gaps.
3Adaptability or versatility
If TBE systems require users to provide paired input/output examples, then transformation programs can be synthesized, but system complexity and ease of use deteriorate
Solution Approach 1:
The system uses copying by learning transformation patterns from historical data pairs and replicating these patterns for new transformation tasks. Instead of requiring users to create new transformation logic, the system copies and adapts proven transformation patterns from the training data to new source-target data pairs.
4Extent of automation
If automatic pattern-based transformation is implemented, then user intervention is reduced, but the need for pre-generated programs and system infrastructure increases
Solution Approach 1:
The system merges multiple functions into a unified pattern-based transformation framework. The same pattern-matching infrastructure handles diverse transformation tasks by learning general patterns from data pairs, combining pattern recognition, program generation, and data transformation into a single integrated system.
Data Source
Figure 1
Figure 2
Figure 3A~3B
AI summary
A transform-by-pattern (TBP) system is configured to proactively suggest relevant TBP programs based on inputted source dataset and target dataset without requiring users typing in examples. The TBP system has access to multiple TBP programs, each of which includes a combination of a source pattern, a target pattern, and a transformation program that is configured to transform data that fits into the target pattern into data that fits into the source pattern. When a source dataset and a target dataset are received from a user, the TBP system identifies a subset of the source dataset and a subset of the target dataset as related data. The TBP system then identifies one or more applicable TBP programs amongst the multiple TBP programs, and suggest or apply at least one of the one or more applicable TBP programs.