Knowledge Graph Data Integration Through Modular ETL Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current approaches for integrating data from heterogeneous data sources into a knowledge graph storage unit are complex, requiring expertise in both data source systems and knowledge graphs, and lack support for large-scale data processing frameworks, with transformation steps often being non-modular and difficult to maintain.
Innovation Solution
A method and system that decouples data transformation from mapping by using an ETL data pipeline framework with modules for receiving, extraction, transformation, mapping, and loading, employing JSON-based mapping notation and tools like Pandas and Apache Spark for efficient in-memory processing, enabling scalable and fault-tolerant integration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data transformation and mapping are combined in a single process, then the integration process is simpler, but the process becomes difficult to maintain and requires expertise in both data source systems and knowledge graphs
Solution Approach 1:
The patent divides the integration process into separate ETL (extraction, transformation, loading) and mapping modules. The ETL process handles data extraction from source systems and transformation into intermediate formats, while the mapping module separately handles the conversion to knowledge graph representations. This segmentation allows each module to be maintained independently by specialists in their respective domains, reducing the need for comprehensive expertise in all areas.
Solution Approach 2:
The patent introduces an intermediate data representation format that serves as a bridge between the source data and the knowledge graph target. The ETL process transforms source data into this intermediate format, and then the mapping module converts it to the final knowledge graph representation. This intermediary layer decouples the complexity of source system variations from the knowledge graph mapping requirements, making the overall process more maintainable.
2Adaptability or versatility
If transformation steps are integrated into mapping rules, then the mapping process is more comprehensive, but the code becomes non-modular and difficult to maintain
Solution Approach 1:
The patent separates transformation logic into the ETL module and mapping logic into the mapping module, with each having its own configurable parameters and processing steps. Transformation operations (filtering, aggregation, format conversion) are defined independently from mapping rules (ontology alignment, property mapping). This modular structure allows each component to be developed, tested, and maintained independently while maintaining comprehensive data processing capabilities.
3Device complexity
If a single-module approach is used for data integration, then the system is simpler, but it lacks support for large-scale data processing frameworks
Solution Approach 1:
The patent designs the ETL and mapping modules with universal interfaces that can work with multiple data source types and target knowledge graph systems. The architecture supports integration with large-scale data processing frameworks like Apache Spark and Hadoop through standardized connectors and processing pipelines. This universality allows the system to handle both small and large-scale data processing requirements without fundamental architectural changes.
Data Source
AI summary
A method for integrating data from different data sources into a knowledge graph storage unit including:transferring data from different data sources into a receiving and extraction module of an extraction-transformation-loading, ETL, data pipeline framework;extracting the loaded data using an extraction data frame and transferring the extracted data to a transformation module;transforming the extracted data using transformation functionalities and transferring the transformed data to a mapping module;mapping the transformed data using mapping rule functionalities and transferring the mapped data to a post-processing module;post-processing the mapped data using a library and transferring the post-processed data to a loading module; andloading the processed data by the loading module to the knowledge graph storage unit;wherein the mapping module uses a JSON-based mapping notation to convert the transformed data into a serialized form, is provided.