Dynamic Data Product Creation via K-Partite Metadata Graph
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data product systems rely on static assets and metadata, failing to dynamically update with new information from various data sources and lack version tracking and access control, limiting user access to specific assets.
Innovation Solution
A method for creating dynamic data products through a k-partite metadata graph, which filters and generates a new representation of asset catalogs based on user queries, enabling dynamic updates, version tracking, and access control.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If static assets and metadata are used in data product systems, then system simplicity is maintained, but dynamic updates and version tracking capabilities are lost
Solution Approach 1:
The patent transforms static metadata into a dynamic metadata graph structure that automatically updates when new data arrives. The graph nodes and edges dynamically represent data assets and their relationships, enabling the system to adapt to new information without manual reconfiguration while maintaining structured organization through graph theory principles.
Solution Approach 2:
The metadata graph serves as an intermediary layer between raw data assets and the data product system. This graph structure mediates between incoming data and the product catalog, automatically processing and integrating new information through graph operations without requiring direct system reconfiguration.
2Loss of information
If comprehensive asset catalogs are maintained, then complete information availability is achieved, but access control and user-specific view management become difficult
Solution Approach 1:
The patent applies local quality by providing different users with customized views of the metadata graph based on their roles and permissions. Each user sees a filtered subset of the complete graph that is relevant to their function, while the underlying complete graph remains intact. This allows comprehensive information storage with selective access presentation.
Solution Approach 2:
The metadata graph is segmented into multiple view layers corresponding to different user types (e.g., administrators, analysts, end users). Each segment presents appropriate levels of detail and access, while the complete graph maintains all information. This segmentation enables simultaneous comprehensive storage and controlled access.
3Productivity
If manual data product creation processes are used, then data accuracy is maintained, but productivity and response time to new information are reduced
Solution Approach 1:
The system implements self-service through automated metadata graph processing. When new data assets arrive, the system automatically updates the metadata graph, identifies affected data products, and generates updated products without manual intervention. This self-service mechanism maintains accuracy through consistent automated processes while dramatically improving productivity.
Solution Approach 2:
The metadata graph structure provides continuous feedback about data relationships and dependencies. When new information is added, the graph automatically propagates updates through connected nodes and edges, triggering appropriate data product regenerations. This feedback loop ensures accuracy is maintained through systematic update propagation rather than manual verification.
Data Source
AI summary
A method and system for dynamic data product creation. A data product may refer to a collection of one or more datasets to which versioning, a set of policies, and/or a set of constraints may be attached. Existing systems/solutions offering data products today rely on the availability of static assets supported by static metadata descriptive thereof. As an improvement over said existing systems/solutions, embodiments disclosed herein enable newly introduced and ingested information, from across various data sources, to update any relevant data product(s) accordingly. Further, versions of any data products may be tracked and be made readily available to users seeking to reproduce work contingent on certain versions of one or more assets that may have been used to originally produce said work at a given point-in-time. Moreover, accessibility or inaccessibility to any given asset may depend on the access authority granted to the user(s) seeking said given asset.