Anomalous Transformation Detection Using Column Profiles

Solution Overview

Problem

Existing data platforms lack effective mechanisms to identify anomalous transformations in data processing, which can lead to data inconsistencies and errors.

Innovation Solution

The method involves generating column profiles based on lineage data and glossary data, comparing new transformations to these profiles, and outputting alerts for anomalous transformations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If data transformations are performed on a data platform, then data processing capability is improved, but data consistency and quality deteriorate due to undetected anomalies

Engineering Contradiction:

Improvedata processing capabilityVSAvoiddata consistency

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system performs preliminary actions by generating column profiles from historical lineage data and glossary information before new transformations are applied. These profiles establish expected patterns of column behavior, data types, and relationships, enabling the system to detect anomalies before they cause data consistency issues. The proactive creation of baseline profiles allows for preventive detection rather than reactive correction.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system implements feedback by continuously comparing new transformation operations against established column profiles and providing alerts when anomalies are detected. This feedback mechanism monitors data transformations in real-time, compares actual transformations with expected patterns from profiles, and notifies users of deviations that may indicate data quality issues, enabling corrective action before consistency is compromised.

Inventive Principle:

Principle #23Feedback

2Ease of manufacture

If traditional anomaly detection methods are used, then implementation simplicity is maintained, but detection accuracy deteriorates due to inability to identify anomalous transformations

Engineering Contradiction:

Improveimplementation simplicityVSAvoidanomaly detection accuracy

Core Design Contradiction:

Ease of manufactureVSMeasurement precision

Solution Approach 1:

The system performs preliminary actions by generating column profiles from historical lineage data and glossary information before new transformations are applied. These profiles establish expected patterns of column behavior, data types, and relationships, enabling the system to detect anomalies before they cause data consistency issues. The proactive creation of baseline profiles allows for preventive detection rather than reactive correction.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system implements feedback by continuously comparing new transformation operations against established column profiles and providing alerts when anomalies are detected. This feedback mechanism monitors data transformations in real-time, compares actual transformations with expected patterns from profiles, and notifies users of deviations that may indicate data quality issues, enabling corrective action before consistency is compromised.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS12326852B2Identifying anomalous transformations using lineage data

Publication Date: 2025.06.10 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

Methods, systems, and computer program products for identifying anomalous transformations using lineage data are provided herein. A computer-implemented method includes generating a set of column profiles for a corresponding set of columns within one or more datasets based at least in part on lineage data and glossary data, wherein the lineage data comprises information related to transformations performed on each column in the set by a computing platform, and wherein the glossary data comprises information related to one or more terms assigned to one or more of the columns; obtaining information related to a new transformation involving at least one column in the set of columns; comparing the new transformation to the set of column profiles to determine whether the new transformation is anomalous; and in response to determining the new transformation is anomalous, outputting an alert to a user of the computing platform.