Graph Model Metadata for Data Lake Queries Without ETL Copies

Solution Overview

Problem

Conventional approaches to creating graph models require resource-intensive extract, transform, and load (ETL) processes and additional storage for copying data, leading to inefficiencies and multiple data copies that need management.

Innovation Solution

A method and device for creating a graph model directly from data stored in a common data format, such as parquet, without copying the data to a separate graph database, using metadata to define vertices and edges, and executing queries or algorithms directly on the data lake.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If ETL processes are used to construct a graph database from source data, then graph queries can be executed on the constructed graph database, but processing resources and storage resources are consumed and multiple copies of data need to be managed

Engineering Contradiction:

Improvegraph query capabilityVSAvoidprocessing resources

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The patent creates a virtual copy of the graph database in memory that mirrors the structure and relationships of the source data without physically duplicating the data. This virtual graph model allows graph queries to be executed on the copy while the original data remains in its compressed columnar format in the data lake, eliminating the need for ETL processes and physical data copying to a separate graph database system

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The patent introduces a memory-resident graph model as an intermediary layer between the query system and the compressed columnar data in the data lake. This intermediary virtual graph structure enables efficient graph query execution by pre-computing and storing the graph topology (vertices, edges, and relationships) in memory, while the actual data remains stored in the optimized columnar format without requiring transformation or duplication

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If ETL processes are used to construct a graph database from source data, then graph queries can be executed on the constructed graph database, but additional storage resources are consumed to store the copy of the source data

Engineering Contradiction:

Improvegraph query capabilityVSAvoidstorage resources

Core Design Contradiction:

ReliabilityVSVolume of stationary object

Solution Approach 1:

The patent creates a virtual copy of the graph database in memory that mirrors the structure and relationships of the source data without physically duplicating the data. This virtual graph model allows graph queries to be executed on the copy while the original data remains in its compressed columnar format in the data lake, eliminating the need for ETL processes and physical data copying to a separate graph database system

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The patent changes the state of the graph data from physical storage to virtual representation. By storing only the graph topology metadata (vertices, edges, relationships) in memory rather than physical data copies, the system achieves graph query capability without the storage overhead of duplicating the actual data, effectively changing the parameter from physical data volume to virtual structure representation

Inventive Principle:

Principle #35Parameter changes

3Productivity

If data is copied to a separate graph database, then graph queries can be executed efficiently, but multiple copies of the data need to be managed

Engineering Contradiction:

Improvegraph query executionVSAvoiddata management complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent creates a virtual copy of the graph database in memory that mirrors the structure and relationships of the source data without physically duplicating the data. This virtual graph model allows graph queries to be executed on the copy while the original data remains in its compressed columnar format in the data lake, eliminating the need for ETL processes and physical data copying to a separate graph database system

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The patent makes the data lake system multi-functional by enabling it to serve both as a compressed columnar storage system and as a graph database system simultaneously. The virtual graph model in memory allows the same data to be queried using both traditional columnar query methods and graph query methods, eliminating the need for separate graph database infrastructure and simplifying data management

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS20250371001A1Techniques for building or querying graph model of data stored in a common data format

Publication Date: 2025.12.04 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

Described are examples for creating a graph model for data obtained from multiple data sources and stored in a common data format. A selection of columns of data stored in the common data format that are to be associated with one another in the graph model can be obtained. Based on the selection, a graph model can be created including metadata that defines a list of vertices corresponding to at least a portion of the columns and a list of edges between two or more of the vertices identifying a relationship between corresponding columns. A graph query or graph algorithm can be performed over the graph model of the data as stored in the common data format without requiring copying of the data to a separate graph database.