Metadata Processing Apparatus for ETL ID Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
ETL tools are designed primarily for processing real data, leading to issues where real data becomes unlinked from its associated metadata due to changes in ID, causing difficulties in maintaining accurate data analysis and understanding.
Innovation Solution
A data processing apparatus and method that ensures real data and metadata remain linked by processing metadata to include the same ID as the real data output from the ETL tool, and synchronizing IDs when changes occur, allowing for consistent and accurate data analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If ETL tool processes real data and changes its ID, then real data can be transformed and loaded, but real data becomes unlinked from metadata
Solution Approach 1:
The patent introduces a metadata processing apparatus as an intermediary component that works in conjunction with the ETL tool. This apparatus receives metadata corresponding to real data being processed, transforms the metadata's identification to match the transformed real data's identification, and outputs the transformed metadata. This intermediary mechanism ensures that the linkage between real data and metadata is maintained throughout the ETL process without interfering with the core data transformation functionality.
Solution Approach 2:
The patent applies preliminary action by transforming the metadata's identification before the real data is loaded. The metadata processing apparatus pre-processes the metadata to anticipate the ID changes that will occur during ETL transformation. By updating the metadata identification in advance to match the expected transformed real data ID, the system ensures seamless linkage is maintained without requiring post-processing or creating linkage conflicts.
2Device complexity
If ETL tool is designed for processing real data only, then processing logic is simple, but metadata cannot be processed to maintain ID consistency
Solution Approach 1:
The patent segments the data processing system into two distinct but coordinated components: the ETL tool for processing real data and a separate metadata processing apparatus for transforming metadata. This segmentation allows each component to maintain its specialized function and simplicity while working together to achieve the overall goal of maintaining ID consistency. The metadata processing apparatus is further segmented into distinct modules for receiving metadata, transforming identification, and outputting transformed metadata.
Solution Approach 2:
The metadata processing apparatus is designed with multi-functionality to handle various metadata transformation scenarios. It can process different types of metadata, adapt to different ETL transformation rules, and maintain compatibility with existing ETL tools. This universal design allows the system to preserve metadata association information across diverse data processing operations without requiring complex specialized processing for each case.
Data Source
AI summary
A data processing apparatus according to one or more embodiments may process metadata indicative of an attribute of real data input into an extract, transform, and load (ETL) tool external to the data processing apparatus. The real data and the metadata may each include an identification (ID). The data processing apparatus may include a memory and a processor. The memory may store a program. The processor may execute the program. The program may cause the processor to process the metadata to include the same ID as the real data output from the ETL tool.