Data Conversion Engine Version Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems face challenges in accurately tracking data through multiple data sources and conversions, particularly at the row and element level, due to manual recording of metadata which can become out of sync with datasets, and the lack of automated revision management for data conversion metadata.
Innovation Solution
A system that automatically creates conversion programs based on conversion metadata, applies version numbers to metadata revisions, and attaches version information to resultant data, ensuring that processing metadata is tied to the data through matching version IDs, stored in a database for easy access and tracking.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If manual recording of metadata is used to track data through multiple data sources and conversions, then data tracking capability is provided, but metadata becomes out of sync with datasets and tracking accuracy deteriorates
Solution Approach 1:
The system enables self-service automation where the metadata recording process serves itself through automated program generation. The system automatically creates conversion programs from conversion metadata, executes them to transform source datasets into target datasets, and then automatically records the execution results as new metadata in the database, eliminating the need for manual intervention and ensuring continuous synchronization between metadata and datasets.
Solution Approach 2:
The system implements feedback mechanisms where the execution results of conversion programs are automatically captured and fed back into the metadata database. This feedback loop ensures that the conversion metadata accurately reflects the actual transformations performed on the datasets, maintaining synchronization and enabling precise tracking of data lineage through multiple conversions and data sources.
2Productivity
If automated program creation from conversion metadata is implemented, then data conversion efficiency is improved, but system complexity increases
Solution Approach 1:
The system applies universality by creating a multi-functional automated program generation mechanism that handles multiple data sources, various conversion operations, and diverse target formats through a single unified approach. The conversion metadata serves as a universal template that can be automatically transformed into executable conversion programs, enabling efficient data transformation across different scenarios without requiring separate manual programming for each case.
3Measurement precision
If version numbers are applied to metadata revisions and attached to resultant data, then data tracking accuracy is improved, but data processing time increases
Solution Approach 1:
The system applies preliminary action by automatically generating version numbers for metadata revisions during the program creation phase, before the actual data conversion executes. This preliminary versioning allows the system to track and identify specific metadata versions without adding significant overhead during the time-critical data conversion process, thus maintaining metadata integrity while minimizing processing time impact.
Data Source
AI summary
A networked computing system where data from multiple sources is automatically tracked through a plurality of conversion steps at the data row and element level. The system allows input of data conversion metadata. The system automatically applies a version number to the conversion metadata. The system includes a data conversion engine. The data conversion engine ensures that processing metadata versions are in sync with versions of converted data. The conversion engine creates version synchronization at the data row and element level thus allowing data tracking at the data row and element level. The system includes a database that stores conversion metadata and execution attributes and a graphical user interface that allows the user access to the conversion metadata.