Semantic Middleware for Automated Data Harmonization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The integration of heterogeneous data sources in IT systems is complex due to differing data formats and technical implementations, leading to issues with data interoperability, agility, and the need for manual data model management, which becomes impractical with the rapid changes in data and data models, especially in the context of 'Big Data'.
Innovation Solution
The use of semantic middleware for data integration and content-related analysis allows for the automatic harmonization of data models by synchronizing data from various sources, transforming formats, and analyzing content semantically to create a Uniform Information Model, enabling seamless data access and usage without knowledge of original formats or sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual modeling of data models and mappings is used, then data consistency can be maintained, but the complexity and time required increase significantly with rapid data changes
Solution Approach 1:
The system performs automatic data harmonization without manual intervention. The harmonization engine automatically discovers data sources, extracts data models, identifies mappings between heterogeneous data formats, and maintains consistency through self-learning mechanisms, eliminating the need for continuous manual modeling and mapping updates
Solution Approach 2:
Manual mechanical processes of data modeling and mapping are replaced by an automated computational system. The harmonization engine uses algorithms to automatically analyze data structures, infer relationships, and generate mappings, substituting human manual work with automated intelligence
2Measurement precision
If classical data processing methods are used, then data accuracy can be maintained, but productivity decreases due to high complexity
Solution Approach 1:
The complex data harmonization process is divided into independent modular components: data source connection module, data extraction module, data model discovery module, mapping generation module, and validation module. Each module handles a specific aspect of the process, allowing parallel processing and improving overall throughput while maintaining accuracy through specialized processing at each stage
Solution Approach 2:
A uniform information model serves as an intermediary layer between heterogeneous data sources and target applications. This intermediate representation standardizes data from different sources while preserving semantic meaning, enabling efficient processing without sacrificing accuracy through automated mapping to the standard model
3Adaptability or versatility
If interfaces are adapted for each new data source, then data interoperability can be achieved, but device complexity increases continuously
Solution Approach 1:
The harmonization engine provides a universal interface that can connect to multiple heterogeneous data sources through standardized protocols. Rather than requiring custom interfaces for each source, the system uses a single adaptable harmonization layer that automatically discovers and adapts to different data source types, maintaining interoperability while preventing complexity accumulation
Solution Approach 2:
The uniform information model acts as a mediator between diverse data sources and target systems. It provides a standardized intermediate representation that decouples source-specific interfaces from target requirements, allowing new data sources to be integrated through the same harmonization process without increasing overall system complexity
4Adaptability or versatility
If data is transformed into common interim representation, then data integration is enabled, but data loss may occur due to format limitations
Solution Approach 1:
The system dynamically adjusts transformation parameters based on the specific characteristics of each data source and target requirements. Rather than using fixed transformation rules, the harmonization engine learns optimal transformation parameters for different data types and contexts, preserving data completeness while enabling integration through adaptive parameter adjustment
Data Source
Figure 1a
Figure 1b
Figure 2a
AI summary
The present invention relates to a method and a system for automated harmonisation of data that are present in different formats and/or of data models from various heterogeneous data sources or databases, using semantic middleware for data integration and content-oriented data analysis, wherein data from connected data sources are synchronised to the middleware, and the content of said data is subjected to semantic analysis and their semantic typing and designations for attributes are harmonised as meta data, preferably such that the superordinate abstract data model of the integrated data is incrementally extended and harmonised.