Automated Data Import Process Adapting to Schema Changes

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current data transfer systems, particularly from relational databases to analytics or data mining systems, require manual labor and are prone to errors due to hard-coded mechanisms that become unreliable with evolving database schemas, leading to inefficiencies and errors in data import processes.

Innovation Solution

An automated data import method that detects changes in the source data schema, modifies the import process accordingly, and executes the updated process to ensure seamless data transfer between different data storage systems, using metadata comparison and script regeneration to maintain data integrity.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If manual data import processes are used with hard-coded mechanisms, then initial data transfer can be accomplished, but the system becomes unreliable when database schemas evolve over time

Engineering Contradiction:

Improvedata import reliabilityVSAvoidschema change adaptability

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The patent implements dynamic detection of schema changes by comparing metadata from the source database against stored metadata, enabling the system to automatically adapt when schemas evolve. This dynamic approach replaces static hard-coded mechanisms with a flexible detection and response system that maintains reliability while accommodating schema changes.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system incorporates feedback loops where the automated import process detects schema changes, triggers modifications to import scripts, and validates the changes. This feedback mechanism ensures the system continuously adapts to schema evolution, resolving the contradiction between maintaining reliability and adapting to changes.

Inventive Principle:

Principle #23Feedback

2Productivity

If automated data import processes are implemented, then productivity increases, but complexity of the system increases due to schema detection and modification requirements

Engineering Contradiction:

Improvedata import efficiencyVSAvoidimport process complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system performs self-service by automatically detecting schema changes, generating modified import scripts, and executing the updated import process without requiring manual developer intervention. This self-service capability increases productivity while managing complexity through automation of the previously manual processes.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent implements preliminary actions by storing metadata from the source database before the import process begins. This pre-captured metadata is then used to detect schema changes and generate appropriate import scripts, streamlining the overall process and reducing the complexity of real-time schema analysis during data import.

Inventive Principle:

Principle #10Preliminary action

3Ease of manufacture

If hard-coded data import mechanisms are used, then initial implementation is straightforward, but maintenance effort increases when schemas evolve requiring developer intervention

Engineering Contradiction:

Improveinitial implementation easeVSAvoidmaintenance ease

Core Design Contradiction:

Ease of manufactureVSEase of repair

Solution Approach 1:

The patent replaces the mechanical system of manual schema change handling with an automated computational system. The automated process detects schema changes, generates modified import scripts, and executes them without requiring developer intervention. This substitution maintains ease of initial implementation while dramatically improving maintenance ease by eliminating manual updates.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

4Manufacturing precision

If manual developer intervention is required for schema changes, then data integrity can be carefully verified, but time consumption increases significantly

Engineering Contradiction:

Improvedata import precisionVSAvoiddeveloper intervention time

Core Design Contradiction:

Manufacturing precisionVSLoss of time

Solution Approach 1:

The system incorporates feedback mechanisms that automatically verify data integrity through metadata comparison and script validation. This automated verification maintains manufacturing precision by ensuring data integrity while eliminating the time-consuming manual review process, as the feedback loop continuously monitors and validates the import process.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The automated system performs self-verification of data integrity through metadata comparison and script generation validation. This self-service capability maintains precise data import while eliminating developer intervention time, as the system independently verifies its own operations through automated checks and validations.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS11461294B2System for importing data into a data repository

Publication Date: 2022.10.04 HITACHI VANTARA LLC

AI summary

A computer-implemented method of importing data from a data source to a target storage system is disclosed. The method involves executing an automated data import process to retrieve data from the data source and update the target storage system based on the retrieved data. The automated data import process operates in dependence on a source data schema of the data source. The method involves detecting a change in the source data schema, and, in response to detecting the change in the source data schema, modifying the automated data import process. The modified automated data import process can subsequently be used to retrieve further data from the data source and update the target storage system based on the further retrieved data.