Heterogeneous Data View Extraction with Dynamic Schema Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing and maintaining a consistent and singular view of data from heterogeneous data sources is challenging due to varying formats, structures, and field names across different software versions and data sources, leading to complexity in data consumption and analysis.
Innovation Solution
A system and method for extracting data views by mapping raw fields from heterogeneous sources to common fields using schema crawlers, metadata, and aggregation data, allowing for consistent data views even when underlying structures change.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is received from multiple heterogeneous data sources with varying formats and structures, then the quantity and diversity of data increases, but the complexity of managing and maintaining consistent data views increases
Solution Approach 1:
The patent introduces a mediator layer between heterogeneous data sources and the data warehouse. This mediator automatically maps fields from various source formats to a unified internal schema, absorbing the complexity of format variations while presenting a consistent data view to consumers. The mapping layer translates diverse external structures into a standardized internal representation without requiring manual intervention for each data source.
Solution Approach 2:
The system dynamically adapts to changing data structures by monitoring schema variations and updating field mappings accordingly. When data formats change due to software version updates or source modifications, the system automatically detects these parameter changes and adjusts the mapping relationships to maintain consistent data views, eliminating the need for manual schema management.
2Measurement precision
If manual mapping of fields between heterogeneous sources and data warehouse is performed, then mapping precision can be controlled, but the time required for data integration increases
Solution Approach 1:
The system performs automatic field mapping without requiring manual configuration for each data source. The mapping mechanism self-adjusts by analyzing field similarities, data types, and semantic relationships between source and target schemas, generating appropriate mappings autonomously. This self-service approach eliminates manual mapping tasks while maintaining high precision through intelligent field matching algorithms.
Solution Approach 2:
The system pre-configures a framework of mapping rules and field relationships that guides the automatic mapping process. Before actual data integration occurs, the system establishes preliminary mapping strategies based on schema analysis and field similarity metrics, enabling rapid and accurate field mapping without time-consuming manual configuration during data integration.
3Adaptability or versatility
If data structures from sources change due to software version updates, then the adaptability of the system to new versions improves, but the difficulty of maintaining consistent data views increases
Solution Approach 1:
The mapping system is designed to be dynamic rather than static, automatically detecting and adapting to schema changes when software versions are updated. The system monitors data structure variations and dynamically adjusts field mappings to accommodate new formats while maintaining consistency with the data warehouse schema. This dynamic adaptation capability allows the system to embrace version changes without increasing operational complexity.
Solution Approach 2:
The system incorporates feedback mechanisms that continuously monitor data structure changes and adjust mappings accordingly. When software version changes cause schema variations, the system receives feedback about these changes through data sampling and schema comparison, then automatically updates the mapping relationships to maintain consistent data views. This closed-loop feedback process eliminates the complexity of manual maintenance by automating the adaptation response.
4Loss of information
If comprehensive field mapping is performed across all data sources, then data completeness improves, but the processing requirements and memory usage increase
Solution Approach 1:
The system performs selective field mapping based on data importance and source characteristics rather than uniformly processing all fields. The mapping mechanism identifies and processes only the most relevant fields for each data source, applying partial mapping strategies that maintain data completeness for critical information while reducing processing overhead for less important fields. This partial action approach balances data completeness with processing efficiency.
Data Source
AI summary
One embodiment includes a method to be performed by a computing system. The method includes receiving first data comprising one or more first data fields associated with corresponding first values, receiving second data comprising one or more second data fields associated with corresponding second values, wherein at least one of the one or more second data fields corresponds to at least one of the one or more first data fields. The method further includes associating one or more of the one or more first data fields and one or more of the one or more second data fields to one or more standard data fields and generating a data view based on the one or more standard data fields.