Unified Workload Automation and Data Lineage Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing workload automation systems lack a comprehensive view of data processing systems, as they separate job dependency information and data lineage information, making it difficult to monitor and adjust workflows efficiently and to determine the impact of changes in either domain.
Innovation Solution
Integrating job dependency information and data lineage information into a common repository, allowing for the combination and analysis of scheduling dependencies with data dependencies, thereby providing a holistic view of the data processing system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If job dependency information and data lineage information are kept separate in different information systems, then each system can maintain its own specialized structure, but a holistic view of the data processing system cannot be obtained and the impact of changes cannot be determined
Solution Approach 1:
The patent merges job dependency information from workload automation systems with data lineage information from data processing systems into a unified information system. This combination allows the system to provide both a holistic view of data processing and the ability to determine impact of changes, resolving the technical contradiction by integrating previously separate information sources.
2Productivity
If workload automation systems only track job dependencies without data lineage information, then scheduling can be simplified, but efficient monitoring and adjustment of workflows becomes difficult
Solution Approach 1:
The unified information system provides feedback mechanisms that combine job dependency status with data lineage information, enabling efficient workflow monitoring and adjustment. The system can determine the impact of changes by analyzing both scheduling dependencies and data dependencies together, improving productivity without excessive operational complexity.
3Loss of information
If data lineage information is analyzed without integrating job dependency information, then data flow can be tracked, but the impact of scheduling changes on data processing cannot be determined
Solution Approach 1:
The system merges data lineage information with job dependency information in a unified information system, enabling comprehensive impact analysis. By combining these previously separate information sources, the system can determine how scheduling changes affect data processing while maintaining manageable system complexity through integrated architecture.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for workload automation and job scheduling information. One of the methods includes obtaining job dependency information, the job dependency information specifying an order of execution of a plurality of jobs. The method also includes obtaining data lineage information that identifies dependency relationships between data stores and transformation, wherein at least one transformation accepts data from a first data store and produces data for a second data store. The method also includes creating links between the job dependency information and the data lineage information. The method also includes determining an impact of a change in a planned execution of an application of the plurality of applications based on the job dependency information, the created links, and the data lineage information.