Metadata Propagation via Generalized ETL Categorization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing metadata propagation methods in data engineering are inflexible and unable to add recalculated metadata, such as statistics, and require rigid rule-based mappings, which limits their applicability and efficiency.
Innovation Solution
A method that categorizes metadata and ETL processes to generalize them, allowing for the definition of propagation rules and generating metadata for ETL-converted data based on these categorizations, enabling flexible metadata propagation and reducing the need for recalculations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If rigid rule-based metadata propagation is used, then metadata extraction is systematic, but metadata extraction time increases and flexibility is reduced
Solution Approach 1:
The system performs preliminary categorization of metadata and ETL processes into general types before actual metadata propagation occurs. By pre-defining these categories and their relationships, the system avoids performing detailed rule matching during each metadata extraction operation, thereby reducing extraction time while maintaining systematic processing through the predefined category framework.
Solution Approach 2:
The patent creates a universal categorization framework where metadata and ETL processes are classified into general types that can be reused across multiple operations. This universal classification system allows the same category definitions to serve multiple metadata propagation tasks, reducing redundant processing and extraction time while ensuring consistent systematic extraction through the unified framework.
2Device complexity
If classification rules are used to determine metadata, then metadata propagation is structured, but adaptability to different metadata types is reduced
Solution Approach 1:
The system segments metadata and ETL processes into distinct categorical types, separating the structure definition from the specific metadata instances. This segmentation allows the structured framework to define general categories while maintaining adaptability by allowing new specific metadata types to be mapped to existing categories without changing the overall structure, thus resolving the contradiction between structured propagation and adaptability.
Solution Approach 2:
The patent enables adaptability by allowing the parameters of metadata categories to be modified and extended. The general type framework accommodates different metadata types by changing the specific parameters and attributes associated with each category, maintaining the structured propagation framework while adapting to diverse metadata requirements through parameter flexibility rather than structural changes.
3Stability of the object's composition
If rule-based metadata determination is used, then metadata consistency is maintained, but ability to add recalculated metadata is lost
Solution Approach 1:
The system introduces an intermediary categorization layer between the structured rule framework and the actual metadata. This intermediary category system acts as a mediator that maintains consistency through the structured framework while allowing flexibility to add recalculated metadata by mapping it to appropriate categories. The category framework serves as the intermediary that reconciles the need for consistency with the ability to add new metadata types including recalculated statistics.
Data Source
AI summary
A method for determining metadata propagation generated by an Extract, Transfer, Load (ETL) process. The method may include categorizing metadata given to data sources to generalize the metadata and categorizing the ETL process to generalize the ETL process; defining propagation rules for the generalized metadata and generalized ETL process; and generating metadata for ETL-converted data based on the categorization of the generalized metadata and generalized ETL process, and corresponding ones of the propagation rules.