Metadata-Driven Data Ingestion for Heterogeneous Schema Evolution

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Efficiently loading heterogeneous sources of data into a data warehouse with constantly evolving schemas is challenging due to schema heterogeneities and the need for dynamic schema management in big data environments like Hadoop MapReduce systems.

Innovation Solution

A meta-data driven approach using a datahub and a generic pipelined data loading framework, which includes configuration files and a catalog, automates data ingestion into HDFS-based data warehouses, enabling efficient handling of heterogeneous data sources and schema evolution without modifying legacy configurations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If a traditional data loading approach is used, then the data warehouse schema can be managed, but the system cannot efficiently handle heterogeneous data sources and schema evolution

Engineering Contradiction:

Improvehandling heterogeneous data sources and schema evolutionVSAvoiddata loading efficiency

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The patent segments the data loading process into distinct Map and Reduce phases, allowing independent processing of heterogeneous data sources. Each data source can be processed through its own mapper function while maintaining a unified reduce phase that handles schema evolution, thus improving adaptability without sacrificing overall loading efficiency.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent implements a universal data loading framework that can handle multiple types of data sources (relational databases, flat files, NoSQL databases) through a common MapReduce interface. This multi-functional approach allows the system to adapt to heterogeneous sources while maintaining efficient parallel processing capabilities.

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Productivity

If a MapReduce framework is used to improve data loading efficiency, then parallel processing capability increases, but the complexity of managing heterogeneous schemas increases

Engineering Contradiction:

Improvedata loading efficiencyVSAvoidschema management complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent introduces an intermediary schema resolution layer between the Map and Reduce phases. This intermediary component automatically resolves schema differences across heterogeneous data sources by applying transformation rules, thus maintaining parallel processing efficiency while reducing the complexity of manual schema management.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent dynamically changes schema parameters during the MapReduce processing by applying transformation configurations that adapt data from different sources to a unified target schema. This parameter-based approach allows efficient parallel processing while automatically managing schema complexity through configurable transformation rules.

Inventive Principle:

Principle #35Parameter changes

3Adaptability or versatility

If legacy configuration files are modified to support new data sources, then adaptability improves, but the risk of breaking existing data loading processes increases

Engineering Contradiction:

Improvesupport for new data sourcesVSAvoidstability of existing data loading processes

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent performs preliminary schema validation and configuration checks before executing data loading processes. By pre-validating new data source configurations against the target schema and existing processing pipelines, the system can integrate new sources while automatically detecting and preventing conflicts that would break existing processes, thus maintaining reliability while improving adaptability.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS8949175B2Meta-data driven data ingestion using MapReduce framework

Publication Date: 2015.02.03 AMOBEE

AI summary

A generic approach for automatically ingesting data into an HDFS (Hadoop File System) based data warehouse includes a datahub server, a generic pipelined data loading framework, and a meta-data model that, together, address data loading efficiency, data source heterogeneities, and data warehouse schema evolvement. The loading efficiency is achieved via the MapReduce scale-out solution. The meta-data model is comprised of configuration files and a catalog. The configuration file is setup per ingestion task. The catalog manages the data warehouse schema. When a scheduled data loading task is executed, the configuration files and the catalog collaboratively drive the datahub server to load the heterogeneous data to their destination schemas automatically.