Historical Data Transformation Programs for Semantic Task Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data transformation systems face challenges in identifying required transformations, providing appropriate samples, and ensuring correct execution, especially for users unfamiliar with datasets, leading to inefficiencies and manual trial-and-error processes.
Innovation Solution
A method that computes similarity scores between a data transformation task and stored transformations using embeddings, forms clusters, and generates a data transformation program from the most similar stored transformations, allowing for automated execution and reuse.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual data transformation processes are used, then flexibility and adaptability are maintained, but productivity and time efficiency deteriorate due to manual trial-and-error
Solution Approach 1:
The system automatically generates data transformation programs by computing similarity scores between task data and stored transformations, and executing the best match without requiring manual user intervention or trial-and-error processes
Solution Approach 2:
The system pre-computes and stores transformation programs with their metadata and similarity scores, so when a transformation task arises, the system can quickly retrieve and execute the pre-prepared transformation without manual intervention
2Adaptability or versatility
If comprehensive stored transformations are maintained for all possible data formats, then adaptability improves, but device complexity and storage requirements worsen
Solution Approach 1:
The system transforms data by changing parameters such as date formats (MM/DD/YYYY to DD/MM/YYYY), data types, and structural representations through computed similarity scores rather than maintaining explicit transformation rules for all possible format combinations
Solution Approach 2:
The similarity computation mechanism serves multiple functions: it measures semantic similarity between data formats, ranks transformation candidates, and guides program generation, making the system universally applicable to diverse data transformation tasks without requiring format-specific logic
3Measurement precision
If exact matching of data formats is required, then transformation accuracy improves, but loss of time worsens due to inability to leverage similar historical examples
Solution Approach 1:
The system copies transformation programs from stored historical examples that have high similarity scores to the current task, adapting proven transformations to new contexts without requiring complete re-verification, thus maintaining accuracy while reducing time loss
Data Source
AI summary
An embodiment computes a plurality of similarity scores, each similarity score in the plurality of similarity scores measuring a similarity of task data of a first data transformation task to data of a stored data transformation in a plurality of stored data transformations, wherein each of the plurality of stored data transformations comprises a stored data transformation program, each stored data transformation program comprising a data transformation from a first data format to a second data format. An embodiment generates, from a first stored data transformation program in the plurality of stored data transformations, a generated data transformation program. An embodiment performs, by modifying a plurality of records described by the first data transformation task into a second plurality of records according to the generated data transformation program, the first data transformation task.