Data Fabric Mediator Enforcing Metadata Lineage Rules
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data fabrics often fail to capture and replicate metadata associated with data items, leading to a lack of contextual information when data is stored and accessed, which is crucial for understanding the data items.
Innovation Solution
A metadata fabric pattern is created that includes rules for metadata collection, data protection, storage, consistency, and coherency, with an identifier appended to the data item to enforce these rules during storage and access, ensuring relevant metadata is captured and stored alongside the data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data replication systems are used to ensure data coherence, then data consistency is improved, but metadata capture and replication is not performed resulting in loss of contextual information
Solution Approach 1:
The patent merges data replication with metadata replication by creating a unified replication process. The system now replicates both data items and their associated metadata together, ensuring that contextual information travels with the data across the fabric. This is achieved through the data fabric mediator that intercepts replication requests and ensures both data and metadata are copied to target locations.
Solution Approach 2:
The system performs preliminary action by capturing metadata at the source before data replication occurs. The data fabric mediator intercepts data items as they are being ingested into the fabric, captures their metadata at this early stage, and stores it in a metadata store. This ensures metadata is available before replication happens, preventing information loss.
2Loss of information
If metadata collection rules are enforced to capture contextual information, then data contextualization is improved, but system complexity increases due to additional metadata collection and enforcement mechanisms
Solution Approach 1:
The patent introduces a data fabric mediator as an intermediary component that sits between data sources and the data fabric. This mediator handles metadata collection, validation, and enforcement centrally, rather than distributing complexity across all system components. The mediator intercepts data items, applies metadata collection rules, and manages the metadata store, thereby reducing overall system complexity while maintaining comprehensive metadata capture.
Solution Approach 2:
The system implements feedback mechanisms where the data fabric mediator continuously monitors data items as they move through the fabric, validates their metadata against defined rules, and triggers appropriate actions. This feedback loop ensures metadata compliance without requiring complex manual intervention, as the system automatically detects and corrects metadata issues.
3Ease of operation
If data items are replicated across multiple storage locations, then data accessibility is improved, but metadata is not replicated resulting in incoherent data views
Solution Approach 1:
The patent combines data replication and metadata replication into a single atomic operation. When data is replicated to multiple storage locations for improved accessibility, the metadata is simultaneously replicated to all target locations. This ensures that every copy of the data has its complete contextual information, maintaining coherence across the distributed fabric.
Solution Approach 2:
The system creates complete copies of both data items and their metadata together when replicating across the fabric. Rather than copying only data or requiring separate metadata copy operations, the system packages and replicates both simultaneously, ensuring that each replicated location receives a complete data package with full contextual information intact.
Data Source
AI summary
Computer-implemented methods for adding lineage data to a data fabric are provided. Aspects include receiving a data item from a data source and obtaining a metadata fabric pattern corresponding to one or more of the data item and the data source. Aspects also include appending an identifier of the metadata fabric pattern to the data item and identifying one or more rules that correspond to the data item based on the identifier. Aspects further include performing one or more actions specified by the one or more rules and storing the data item in one or more data storage devices of the data fabric.