Bootstrapping Data Lakes with Dataset Join Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current big data platforms struggle to leverage existing application patterns embedded in data runtimes, such as SQL and ETL processes, to accurately join diverse data sources, leading to incomplete insights and inefficient data analysis.
Innovation Solution
A method and system to mine relationship patterns from runtime artifacts, including SQL and ETL processes, to bootstrap data lakes with dataset joins metadata, enabling the extraction and classification of relationship clauses and the generation of normalized syntax trees for improved data integration and analytics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional data processing applications are used to process big data, then the processing method is simple and familiar, but the data sets are too large or complex to be processed effectively
Solution Approach 1:
The patent segments the complex data processing task into multiple components: distributed data storage across multiple nodes, parallel query execution, and modular data lake architecture. This allows big data to be processed by breaking it into manageable chunks that can be handled by individual processing units simultaneously.
Solution Approach 2:
The patent introduces a new dimensional approach by creating a data lake layer that sits above traditional relational databases, enabling queries to access and join data across multiple sources without moving the underlying data. This adds a new dimension to data access and processing architecture.
2Adaptability or versatility
If data is stored in separate silos, then data storage is simple and organized, but new analytics and querying capabilities across diverse datasets cannot be achieved
Solution Approach 1:
The patent creates a universal data lake architecture that can handle multiple types of data sources (relational databases, flat files, XML, JSON, sensor data) through a common interface and query mechanism, enabling diverse analytics capabilities across previously siloed data.
Solution Approach 2:
The patent introduces an intermediary layer (the data lake) that mediates between diverse data sources and query operations. This intermediary provides standardized access methods and join capabilities without requiring data movement or complex point-to-point integrations between data silos.
3Loss of information
If data manipulation statements are not analyzed, then runtime artifacts remain unprocessed, but relationship patterns between datasets cannot be discovered
Solution Approach 1:
The patent performs preliminary analysis of data manipulation statements (SQL queries, ETL processes) to extract relationship patterns between datasets before actual data querying occurs. This pre-extraction of metadata and relationship information enables more efficient subsequent data access and analytics.
Data Source
AI summary
A method to search for at least one relationship pattern in a plurality of runtime artifacts is provided. The method may include detecting at least one data manipulation statement in the plurality of runtime artifacts. The method may also include extracting at least one relationship clause from the detected at least one data manipulation statement. The method may further include parsing the extracted at least one relationship clause. The method may include generating at least one normalized syntax tree based on the parsed at least one relationship clause. The method may also include performing a classification and a snippet discovery on the generated at least one normalized syntax tree.