Enterprise Data Mesh Schema Mapping for Heterogeneous Integration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The incompatibility of different data model schemas used by various entities within an organization, resulting from mergers and acquisitions, prevents the integration and collective analysis of data, hindering efficient access, retrieval, and compliance across the organization.
Innovation Solution
A data management system that defines a uniform enterprise data model schema, integrates data from diverse sources, and transforms it into a common format, enabling unified access and analysis across the organization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If different data model schemas are used by various entities, then each entity can maintain its own data structure and access patterns, but data integration and collective analysis across entities become impossible
Solution Approach 1:
The patent introduces a data mesh as an intermediary layer between entities with different data models and the central data lake. This data mesh contains schema mappings that translate between heterogeneous data models, enabling data integration without requiring entities to abandon their own data structures. The data mesh acts as a mediator that preserves entity autonomy while enabling collective data analysis.
2Productivity
If data from multiple entities with different data models is integrated, then collective analysis and organization-wide insights can be achieved, but data quality issues and inconsistencies arise
Solution Approach 1:
The patent implements data quality control mechanisms that perform preliminary validation, cleaning, and standardization of data during the ingestion process into the data lake. Schema mappings in the data mesh pre-process data to ensure consistency before integration, preventing data quality issues from propagating through the system. This preliminary action ensures that collective data analysis relies on reliable, consistent data.
3Ease of operation
If a uniform enterprise data model schema is imposed on all entities, then data integration and access become simplified, but entities lose the ability to maintain their own data structures and access patterns
Solution Approach 1:
The patent segments the data architecture into multiple layers: entity-specific data models, a translation layer (data mesh with schema mappings), and a unified data lake. This segmentation allows each entity to maintain its own data structure while the translation layer handles the complexity of unification. Entities interact with their familiar data models, while the system provides unified access through the data mesh, eliminating the need to impose a uniform schema on all entities.
Data Source
AI summary
Methods and systems are presented for collectively storing, managing, and analyzing data associated with different data sources. A data management system defines an enterprise data model schema based on different data model schemas associated with the different data sources. The data management system generates, for each data source, an enterprise data model instance based on the enterprise data model schema. Data is ingested from the different data sources, and then transformed and stored in a corresponding enterprise data model instance based on a mapping between a corresponding data model schema and the enterprise data model schema. Upon ingesting the data from the data sources, one or more consolidated data views are generated that combine at least portions of data from different enterprise data model instances. The data arranged according to the one or more consolidated data views is presented on a device and/or further analyzed to produce an analysis outcome.