Unified Metadata Graph via RAG Framework
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in accessing and managing siloed data across disparate locations due to the complexity of data silos, leading to inefficiencies in data retrieval, wasted computational resources, and potential data integrity issues, as each data silo has unique configurations and lacks a unified interface for search operations.
Innovation Solution
A unified metadata graph system that uses natural language processing and Large Language Models (LLMs) to determine semantically similar metadata identifiers, generating a domain-specific metadata graph that reduces data retrieval times and preserves data integrity by providing an abstract layer for data location and lineage, thereby eliminating the need for creating new data silos or reconfiguring existing systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data silos are created for each computing system to efficiently process data, then data processing efficiency is improved, but compute capacity requirements increase due to reading and maintaining multiple copies of data
Solution Approach 1:
The patent introduces a metadata graph as an intermediary layer between data silos and computing systems. This metadata graph stores abstracted data location information and lineage relationships, allowing systems to locate and access data across silos without creating physical copies. The metadata graph acts as a mediator that resolves data location queries and maintains data integrity without requiring redundant data storage.
Solution Approach 2:
The patent segments data access by separating data storage locations from data location information. Instead of storing actual data copies in multiple silos, the system stores metadata segments that describe where data resides and how to access it. This segmentation allows efficient data processing while reducing compute capacity requirements by eliminating redundant data copies.
2Reliability
If each computing system has its own data silo with unique variable names and sequencing keys, then data integrity is maintained, but data retrieval complexity increases due to lack of unified interface
Solution Approach 1:
The patent creates a universal metadata graph interface that serves multiple functions: locating data across different silos, understanding data lineage, and resolving variable name differences. This single interface replaces the need for system-specific query mechanisms, providing unified access while preserving the integrity maintained by each silo's unique configuration.
Solution Approach 2:
The metadata graph serves as an intermediary layer that translates between different silo-specific data formats and a unified query interface. It maintains the integrity of each silo's unique variable names and sequencing keys while providing a consistent way to access data across all silos, thereby reducing retrieval complexity without compromising data integrity.
3Adaptability or versatility
If copies of data are maintained in different data silos for different computing systems, then data availability is improved, but computational resources are wasted reading and maintaining single version of truth
Solution Approach 1:
The metadata graph acts as an intermediary that enables data availability across silos without creating physical copies. It stores references to original data locations and maintains lineage information, allowing systems to access data from its original source while preserving availability. This eliminates the need for redundant data copies and the computational resources required to maintain them.
Solution Approach 2:
The patent adds a metadata dimension to data storage, separating the physical data from its location information. Instead of storing data copies across multiple silos, the system stores data once and creates metadata references that point to it from multiple locations. This dimensional separation maintains data availability while eliminating redundant storage and the computational overhead of maintaining multiple versions.
4Productivity
If new applications are built using latest technologies and techniques, then application performance is improved, but system complexity increases over time as technologies become obsolete
Solution Approach 1:
The patent implements a dynamic metadata graph that automatically updates as data moves between silos and as new computing systems are added. The metadata graph adapts to changing system configurations and technologies without requiring manual reconfiguration, allowing applications to use latest technologies while maintaining manageable system complexity through automated metadata management.
Data Source
AI summary
Systems and methods for reducing data retrieval times when accessing siloed data across disparate locations by generating a unified metadata graph via a Retrieval-Augmented Generation (RAG) framework are disclosed. The system selects a first Large Language Model (LLM) prompt corresponding to a first metadata identifier of the set of metadata identifiers. The system then augments the first LLM prompt with the first metadata identifier to be provided to an LLM, where the LLM is configured to generate a first intermediate output. The system then augments the first LLM prompt with the second set of metadata identifiers corresponding to the first metadata identifier to be provided to the LLM, where the LLM generates a second intermediate output. The system then generates a domain-specific unified metadata graph. Responsive to determining that first performance metrics fail to satisfy a performance criteria, the system performs an update process on the domain-specific unified metadata graph.