Load Plan Generation via Metadata Merging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current automated load plan generation tools require separate repositories and synchronization, leading to inefficiencies in data integration scenarios and increased manual effort for loading data into data warehouses.
Innovation Solution
A method for generating load plans that configures predefined components based on design and runtime dependencies, allowing for automated orchestration of data loading tasks from various sources into data warehouses, reducing the need for manual intervention and separate repositories.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If automated load plan generation tools are used, then productivity is improved, but device complexity increases due to requiring separate repositories and synchronization mechanisms
Solution Approach 1:
The patent merges the load plan generation functionality with the existing data warehouse metadata repositories, eliminating the need for separate repositories. The load plan generation tool directly utilizes metadata from existing repositories (data source definitions, fact table definitions, transformation definitions) to generate execution plans, thereby removing the complexity of maintaining and synchronizing separate repository systems while preserving automated generation capabilities
Solution Approach 2:
The system makes existing metadata repositories serve dual purposes: both for data warehouse development and for automated load plan generation. By designing the load plan generation to consume metadata from existing repositories and allowing those same repositories to store generation results, the system achieves multi-functionality without requiring dedicated separate infrastructure, thus improving productivity without adding repository synchronization complexity
2Device complexity
If manual load plan creation is used, then device complexity is reduced, but productivity decreases due to increased manual effort
Solution Approach 1:
The system implements self-service automation where the load plan generation tool automatically consumes metadata from existing repositories (data sources, fact tables, transformations) and generates executable load plans without manual intervention. This allows the system to serve itself by utilizing its own existing metadata infrastructure, thereby maintaining architectural simplicity while dramatically improving data loading productivity through automated plan generation and execution
3Reliability
If separate repositories are used for load plan generation, then reliability is improved through dedicated metadata storage, but loss of time increases due to synchronization requirements
Solution Approach 1:
The patent combines load plan generation metadata storage with existing data warehouse repositories, eliminating separate repository infrastructure. By having the load plan generation process read from and write to the same repositories used for data warehouse development, the system maintains metadata reliability through established repository mechanisms while completely eliminating synchronization delays, as no separate repository synchronization is required
Data Source
AI summary
Load Plan Generator (LPG) is a BIAPPS utility for generating ODI load plans based on desired subset of fact tables for loading BIAPPS Data Warehouse. The tool simplifies the configurations process by minimizing the manual steps and configurations and provides a guided list of configurations steps and checklists. The load plan components can include different sets of load plans that will be stitched together by the load plan generator to create one load plan for loading chosen fact groups in the warehouse sourcing from different transaction systems.