Data Lake Graph Modeling with Virtual Vertices for M:M Relationships
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional graph model construction methods are resource-intensive and require extensive human engineering, leading to multiple data copies that need management for accuracy, privacy, and security, especially when dealing with complex many-to-many (m:m) relationships.
Innovation Solution
A method to infer a graph model directly from data in a data lake by detecting joinable columns, creating virtual vertex tables to break m:m relationships into m:1 relationships, and labeling vertices and edges with primary keys and foreign-key column names, without requiring a separate graph database.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If ETL processes are used to construct a graph database by copying source data, then a graph model can be created for data analysis, but computing resources and storage resources are consumed excessively
Solution Approach 1:
The patent applies a virtualization-based copying approach where virtual vertex tables and virtual edge tables are created as references to source data in the data lake, rather than physically copying data. The graph model consists of virtual structures that point to original data locations, enabling graph queries without duplicating data storage or requiring extensive ETL processing.
2Productivity
If ETL processes are used to construct a graph database, then a graph model can be created, but human resources for designing the graph database are consumed excessively
Solution Approach 1:
The system performs self-service by automatically inferring the graph model structure from the source data schema in the data lake. The virtual graph model is constructed by automatically identifying vertex tables, edge tables, and their relationships based on the existing data lake schema, eliminating the need for manual graph database design and reducing human resource requirements.
Solution Approach 2:
The patent performs preliminary action by pre-defining the virtual graph model structure that maps to the existing data lake schema. The system proactively identifies joinable columns, determines relationship types (1:1, 1:m, m:1, m:m), and creates virtual tables before any graph queries are executed, so that the graph model is ready for immediate use without manual intervention.
3Productivity
If ETL processes create multiple copies of data, then a graph database can be constructed, but data accuracy, privacy, and security management becomes complex
Solution Approach 1:
The patent uses virtual copying where virtual vertex tables and virtual edge tables are created as schema-level abstractions that reference source data in the data lake without physical data duplication. This approach maintains a single source of truth while enabling graph model functionality, thereby simplifying data accuracy, privacy, and security management.
4Productivity
If conventional ETL processes are used, then data can be transformed into a graph database, but the process is resource intensive in terms of both computing and storage
Solution Approach 1:
The patent implements virtual copying where the graph model is represented as virtual tables containing references or pointers to source data in the data lake, rather than physical copies. The virtual vertex tables and virtual edge tables store minimal metadata (such as foreign key relationships and join conditions) while the actual data remains in the original data lake storage, dramatically reducing storage resource consumption.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Apparatuses, methods, and computer-readable media infer a graph model from data. The apparatus detects joinable columns within a plurality of columns of source tables in a data lake, the joinable columns defining a plurality of relationships between columns and including columns with an m:m relationship. The apparatus extracts, from the joinable columns, a virtual vertex table represented by a union of corresponding columns that breaks the m:m relationship into two m:1 relationships. The apparatus identifies at least one primary key for each relationship of the plurality of relationships, each relationship defining at least one from-table and one to-table. The apparatus creates a vertex table in the graph model for each source table or virtual vertex table having a primary key. The apparatus creates an edge table in the graph model for each from-table defined in the plurality of relationships.