Database Playback Architecture for Source-Independent Workflows
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In complex and distributed workflow integrated applications, data collection from multiple systems and data sources can lead to performance issues if data is missing or unavailable, causing the workflow to stop.
Innovation Solution
A database playback architecture is implemented, utilizing a data modeling engine to collect data from multiple sources, build a standard data model, and prepare a configuration table to support workflow execution, making the workflow independent of data sources and enhancing scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If data collection is performed from multiple data sources in a distributed workflow, then data completeness is improved, but workflow reliability deteriorates when data is missing or sources are unavailable
Solution Approach 1:
The system performs preliminary data collection from all data sources before workflow execution begins. The data modeling engine retrieves and stores data from multiple sources in advance, creating a pre-populated data repository. This ensures that when the workflow executes, all required data is already available, eliminating interruptions due to missing data or unavailable sources.
Solution Approach 2:
The data modeling engine acts as an intermediary between multiple data sources and the workflow execution. It consolidates data from various sources into a unified data model structure, buffering the workflow from the complexities and unavailability of individual sources. This intermediary layer ensures continuous workflow execution by managing data availability independently.
2Adaptability or versatility
If a standard data model is implemented to support workflow, then workflow independence from data sources is improved, but device complexity increases
Solution Approach 1:
The data modeling engine implements a universal data model that can accommodate multiple data sources with different structures. This standardized model serves multiple purposes: it unifies data from various sources, provides consistent workflow interfaces, and enables flexible configuration without modifying core workflow logic. The universal model abstracts away source-specific complexities while maintaining adaptability.
Solution Approach 2:
The system creates a virtual copy of the data model structure that mirrors the actual data sources. This virtual representation allows the workflow to interact with data as if sources were always available and consistent, without directly coupling to the complex underlying data source infrastructure. The copied model simplifies workflow interactions while preserving data relationships.
3Loss of information
If data is collected at different stages of workflow execution, then data availability is improved, but workflow performance deteriorates due to potential stops
Solution Approach 1:
Instead of collecting data at various stages during workflow execution, the system performs all data collection in advance before the workflow begins. The data modeling engine retrieves data from all sources and stores it in the data model repository beforehand. This eliminates the need for intermittent data collection operations that would cause workflow pauses, ensuring continuous execution and improved performance.
Data Source
AI summary
A computer system includes multiple data sources each configured to store different data for a portion of a distributed workflow integrated application, a data modeling engine, and processor hardware configured to execute instructions to define a standard data model, the standard data model configured to support a workflow, obtain access details for obtaining data from the multiple data sources, set up the data modeling engine to connect to the multiple data sources, based on the obtained access details, prepare a configuration table to return a list of the multiple data sources used in the workflow and an execution order of the multiple data sources, collect, by the data modeling engine, data from the multiple data sources, and prepare a data object using the standard data model and the data collected from the multiple data sources.