Graph Database Metadata Management for Enterprise Data Lineage

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Large enterprises face challenges in efficiently managing and tracking vast amounts of data due to the limitations of existing metadata management tools, which are often costly and rely on relational databases that struggle with scalability for lineage tracking.

Innovation Solution

Implementing a metadata management module that utilizes graph databases to represent structured data movement within an enterprise, enabling transparency into data flow and transformation landscapes, and improving data governance and developer productivity through graph traversal techniques and pattern matching algorithms.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If relational databases are used for metadata management, then data storage and retrieval are achieved, but scalability for lineage tracking deteriorates

Engineering Contradiction:

Improvedata storage reliabilityVSAvoidlineage tracking scalability

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent changes the fundamental data model parameter from relational tables to graph database structure, where data elements are represented as nodes and relationships as edges. This parameter change enables efficient traversal of data lineage paths without the scalability limitations of relational joins, directly resolving the contradiction between reliable data storage and scalable lineage tracking.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent substitutes the mechanical query mechanism of relational databases (SQL joins and table relationships) with graph database traversal mechanisms. This substitution allows for efficient navigation of data lineage through graph algorithms, achieving scalable lineage tracking while maintaining data storage reliability.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Ease of operation

If existing metadata management tools are deployed, then data tracking capability is provided, but cost increases and vendor dependency occurs

Engineering Contradiction:

Improvedata tracking capabilityVSAvoidcost and vendor dependency

Core Design Contradiction:

Ease of operationVSQuantity of substance

Solution Approach 1:

The patent adopts open-source graph database technology (such as Neo4j) instead of proprietary metadata management tools. This choice provides the necessary data tracking capability while eliminating vendor lock-in and reducing costs, as open-source solutions can be freely deployed and modified without licensing fees or vendor dependency.

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

Solution Approach 2:

The patent implements a self-service metadata management system where the organization deploys and manages its own graph database infrastructure. This eliminates dependency on external vendors for tool deployment, maintenance, and updates, while providing full data tracking capability through the custom-built system.

Inventive Principle:

Principle #25Self-service

3Loss of information

If comprehensive metadata extraction is performed across multiple databases, then complete data lineage is achieved, but processing time and computational resources increase

Engineering Contradiction:

Improvedata lineage completenessVSAvoidprocessing time

Core Design Contradiction:

Loss of informationVSLoss of time

Solution Approach 1:

The patent performs preliminary extraction and storage of metadata from multiple databases into the graph database structure. By pre-processing and organizing metadata relationships in the graph format, the system enables rapid lineage queries without requiring time-consuming data extraction and analysis at query time, thus achieving complete lineage information with reduced processing time.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent changes the data organization parameter from distributed database tables to a centralized graph database model. This parameter change allows for efficient storage and traversal of comprehensive metadata relationships, enabling complete data lineage tracking across multiple source databases with optimized processing performance through graph algorithms.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS11941566B2Systems and methods for enterprise metadata management

Publication Date: 2024.03.26 JPMORGAN CHASE BANK NA

AI summary

Various methods, apparatuses/systems, and media for managing metadata are disclosed. A processor extracts technical metadata corresponding to enterprise applications from a plurality of databases; builds a metadata repository in a graph database; builds a web-based metadata application based on developing a normalized representation for data flows corresponding to the extracted technical metadata by utilizing the graph database. The extracted technical metadata is stored onto the metadata repository in the graph database. The processor authenticates and authorizes a user to utilize the web-based metadata application; receives search criteria from the user; accesses the metadata repository in the graph database to retrieve the technical metadata and/or data lineage within the enterprise applications from the metadata repository based on received search criteria; and displays the technical metadata and/or the data lineage within the enterprise applications onto a user interface.