Trihybrid Data Governance System Eliminates Custom Scripts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data transfer systems between different systems are error-prone and resource-intensive due to the need for custom-written scripts, which are cumbersome and complex to manage, especially when considering data governance and provenance requirements.
Innovation Solution
A trihybrid data movement, data governance, and data provenance system that centralizes data transfers, eliminating the need for custom scripts by using a user interface with a distribute module, application programming interfaces, and a metadata store to manage data ownership and lineage, allowing multiple subscribers to access published data sets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If custom-written software scripts are used for each data transfer, then data transfer functionality is achieved, but the system becomes time-consuming and resource-consuming to implement and maintain
Solution Approach 1:
The patent implements a universal data transfer system that can handle multiple data transfer scenarios through a single platform. The system provides standardized interfaces and pre-built capabilities that work across different data sources and destinations, eliminating the need to write custom scripts for each transfer operation. This universal approach directly addresses the productivity improvement while reducing time loss.
Solution Approach 2:
The system enables self-service data transfers through automated scheduling, error handling, and monitoring capabilities. Once configured, the data transfer system operates autonomously without requiring manual script intervention for each transfer, thereby improving productivity and reducing the time required for repeated transfer operations.
2Adaptability or versatility
If custom scripts are written for each data transfer, then specific transfer requirements are met, but the scripts become error-prone and cumbersome to manage
Solution Approach 1:
The patent introduces an intermediary layer between data sources and destinations that handles the complexity of data transfers. This intermediary provides standardized connectors, validation rules, and error handling mechanisms that reduce errors while maintaining adaptability. The intermediary abstracts the complexity away from individual transfer operations, improving reliability without sacrificing customization capability.
Solution Approach 2:
The system incorporates feedback mechanisms through automated monitoring, error reporting, and validation. Transfer operations are tracked and verified, with automatic notifications and corrections for issues detected. This feedback loop significantly reduces errors while maintaining the ability to handle diverse transfer requirements.
3Loss of information
If multiple data governance and data provenance databases are used, then data ownership and lineage tracking are achieved, but the system complexity increases and synchronization becomes difficult
Solution Approach 1:
The patent merges data governance and data provenance tracking into a unified system that operates alongside the data transfer functionality. Instead of maintaining separate databases for governance and provenance information, the system integrates these capabilities into the core transfer mechanism, thereby reducing system complexity while maintaining complete tracking of data ownership and lineage.
Solution Approach 2:
The system performs preliminary actions by automatically capturing and storing governance and provenance metadata at the time of data transfer. This preliminary capture eliminates the need for separate tracking databases and synchronization mechanisms, as all necessary information is recorded upfront during the transfer operation itself.
4Productivity
If custom scripts are used for data transfers, then data movement functionality is achieved, but the system becomes cumbersome and difficult to synchronize with governance requirements
Solution Approach 1:
The patent segments the data transfer system into modular components with clearly defined functions. Each component handles specific aspects of data transfer, governance, or provenance tracking independently but coordinates through standardized interfaces. This segmentation makes the system easier to manage and operate while maintaining high productivity, as each module can be configured and maintained separately.
Data Source
AI summary
A user interface for interfacing with a trihybrid data movement, data governance and data provenance system is provided. The user interface may include an application selection drop-down field that enables selection of an application. The user interface may include a deliveries tab. When selected, the deliveries tab may display the status of user subscription deliveries. The user interface may include a resources tab. When selected, the resources tab may display publications and/or subscriptions that have been set up. When selected, the resources tab may enable a user to create a new publication. The user interface may include an approvals tab. When selected, the approvals tab may display approvals relating to subscriptions that have been requested. The user interface may include a data catalog tab. When selected, the data catalog tab may display a plurality of available publications, and enable a user to request a subscription to a publication.