Data Lake Graph Inference Using Virtual Vertex Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional graph model construction methods are resource-intensive and require extensive human engineering, leading to multiple data copies that need management for accuracy, privacy, and security, especially when dealing with complex many-to-many (m:m) relationships.
Innovation Solution
A method to infer a graph model directly from a data lake by detecting joinable columns, breaking m:m relationships into m:1 relationships using virtual vertex tables, and creating vertex and edge tables with primary keys, without requiring a separate graph database.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If ETL processes are used to construct a graph database by copying source data, then a graph model can be obtained for data analysis, but computing resources and storage resources are significantly consumed
Solution Approach 1:
The patent applies the copying principle in reverse - instead of copying source data into a graph database, it creates a virtual graph model that references the original data. The graph model contains vertex and edge definitions that map to source tables and columns, allowing graph queries without physical data duplication. This eliminates the resource-intensive ETL copying process while maintaining graph analysis capabilities.
Solution Approach 2:
The patent introduces a virtual graph model as an intermediary layer between the data lake and analysis tools. This virtual model acts as a mediator that translates graph queries into SQL queries against the source data, eliminating the need for physical data copying while enabling graph-based data exploration and analysis.
2Reliability
If ETL processes are used to construct a graph database, then a graph model can be obtained, but human resources for designing the graph database are significantly consumed
Solution Approach 1:
The patent implements self-service by automatically inferring the graph model structure from the source data schema. The system automatically detects joinable columns, identifies relationships, and generates vertex and edge table definitions without requiring manual graph database design. This eliminates the need for human resources to design the graph database structure while maintaining accurate data relationships.
Solution Approach 2:
The patent performs preliminary analysis of the source data schema to automatically identify joinable columns and relationships before graph model construction. By pre-processing the schema information and automatically generating the graph model structure, it eliminates manual design effort while ensuring the graph model accurately reflects the underlying data relationships.
3Reliability
If ETL processes create multiple copies of data, then a graph database can be constructed, but data accuracy, privacy, and security management becomes complex
Solution Approach 1:
The patent eliminates physical data copying by creating a virtual graph model that references the original data in the data lake. The graph model contains metadata definitions rather than actual data copies, which maintains data accuracy, privacy, and security while enabling graph analysis functionality.
4Reliability
If conventional ETL processes are used, then a graph database can be constructed, but deployment time and resource consumption increase
Solution Approach 1:
The system automatically infers the graph model structure from the source data schema without requiring manual ETL process design or execution. This self-service approach rapidly generates a functional graph model by automatically detecting relationships and generating vertex and edge definitions, significantly accelerating deployment while maintaining graph analysis functionality.
Solution Approach 2:
The patent performs preliminary automatic analysis of the data schema to identify relationships and generate the graph model structure before deployment. This pre-computation eliminates the time-consuming manual ETL process and enables rapid deployment of functional graph models for immediate data exploration and analysis.
Data Source
AI summary
Apparatuses, methods, and computer-readable media infer a graph model from data. The apparatus detects joinable columns within a plurality of columns of source tables in a data lake, the joinable columns defining a plurality of relationships between columns and including columns with an m:m relationship. The apparatus extracts, from the joinable columns, a virtual vertex table represented by a union of corresponding columns that breaks the m:m relationship into two m:1 relationships. The apparatus identifies at least one primary key for each relationship of the plurality of relationships, each relationship defining at least one from-table and one to-table. The apparatus creates a vertex table in the graph model for each source table or virtual vertex table having a primary key. The apparatus creates an edge table in the graph model for each from-table defined in the plurality of relationships.