Graph Model Metadata for Querying Data Lakes Without ETL
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional approaches to creating graph models require resource-intensive extract, transform, and load (ETL) processes and additional storage for copying data, leading to inefficient use of processing and storage resources.
Innovation Solution
A method and device for creating a graph model directly from data stored in a common data format, such as parquet, without copying the data to a separate graph database, by defining vertices and edges based on metadata, allowing graph queries and algorithms to be executed directly on the data lake.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional ETL processes are used to create a graph database, then graph queries can be executed, but processing resources and storage resources are consumed excessively
Solution Approach 1:
The patent merges the data storage layer (data lake with parquet files) and the graph model layer into a unified architecture. The graph model is stored as metadata within the same storage system that holds the original data, eliminating the need for separate graph database infrastructure and reducing resource consumption while maintaining graph query capabilities.
Solution Approach 2:
Instead of copying entire datasets to create a graph database, the patent creates a lightweight graph model representation that references the original data locations. This selective copying of only the necessary structural information (vertices and edges definitions) reduces storage overhead while enabling graph operations.
2Reliability
If conventional ETL processes are used to create a graph database, then graph queries can be executed, but additional storage resources are required to store the copy of source data
Solution Approach 1:
The patent combines the graph model metadata with the original data storage in the data lake. By storing graph structures as metadata alongside the source parquet files rather than in a separate graph database, the system eliminates redundant storage while preserving full graph query functionality.
Solution Approach 2:
The patent creates a minimal copy of only the graph structural information (vertex and edge definitions) rather than copying the entire source dataset. This lightweight metadata layer references the original data locations, enabling graph operations without duplicating the bulk data storage.
3Reliability
If conventional ETL processes are used to create a graph database, then a graph model can be constructed, but multiple copies of data need to be managed
Solution Approach 1:
The patent merges the graph model management with the existing data lake infrastructure. By using the same storage system and metadata mechanisms that already manage the parquet files, the system eliminates the need for separate graph database management processes, reducing operational complexity while maintaining graph model construction capabilities.
4Productivity
If data is copied to a separate graph database, then graph queries can be executed efficiently, but the architecture becomes more complex
Solution Approach 1:
The patent combines graph query processing with the existing data lake query infrastructure. By implementing graph query capabilities directly on top of the parquet storage system using its existing metadata and query mechanisms, the system maintains query efficiency without requiring a separate graph database architecture.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Described are examples for creating a graph model for data obtained from multiple data sources and stored in a common data format. A selection of columns of data stored in the common data format that are to be associated with one another in the graph model can be obtained. Based on the selection, a graph model can be created including metadata that defines a list of vertices corresponding to at least a portion of the columns and a list of edges between two or more of the vertices identifying a relationship between corresponding columns. A graph query or graph algorithm can be performed over the graph model of the data as stored in the common data format without requiring copying of the data to a separate graph database.