Automated Data Validation Using Mapping Files

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

ETL processes face challenges in accurately transforming data between different formats, leading to errors and inefficiencies, especially when migrating data from legacy systems to new systems, and existing methods lack effective automated validation tools to identify and correct discrepancies.

Innovation Solution

A system and method for automated data validation that uses a mapping file to compare records across files with different attributes and formats, allowing for efficient comparison and error identification by standardizing data and focusing resources on relevant data, enabling parallel processing and reducing computational and memory usage.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If automated data validation compares all attributes in files with different formats, then data accuracy is improved, but computational resources and processing time increase significantly

Engineering Contradiction:

Improvedata validation accuracyVSAvoidcomputational resource consumption

Core Design Contradiction:

Measurement precisionVSUse of energy by moving object

Solution Approach 1:

The patent extracts and compares only the specific attributes identified in the mapping file between source and target files, rather than comparing all attributes. This selective extraction approach validates data accuracy for critical fields while reducing computational overhead by ignoring irrelevant attributes.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent performs partial comparison by focusing on a subset of attributes specified in the mapping file rather than exhaustive comparison of all attributes. This partial action provides sufficient validation for data migration while conserving computational resources.

Inventive Principle:

Principle #16Partial or excessive action

2Reliability

If automated data validation processes large volumes of data with full detail, then validation thoroughness is improved, but processing time increases

Engineering Contradiction:

Improvevalidation thoroughnessVSAvoidprocessing time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent segments the validation process into distinct phases: loading source and target files, reading the mapping file to identify attributes to compare, comparing records based on key attributes, and generating reports. This segmentation allows parallel processing of different file pairs and efficient resource utilization.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent performs preliminary actions by loading entire source and target files into memory at the outset, and by pre-identifying the attributes to compare through the mapping file before actual comparison begins. This preparation enables faster subsequent processing and reduces I/O operations during the comparison phase.

Inventive Principle:

Principle #10Preliminary action

3Measurement precision

If automated data validation uses detailed metadata and file-specific formats, then comparison accuracy is improved, but system complexity increases

Engineering Contradiction:

Improvecomparison accuracyVSAvoidsystem complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent employs a universal mapping file format that can define relationships between attributes in different source and target file formats. This single mapping file structure handles multiple file types and formats without requiring format-specific validation logic, reducing system complexity while maintaining comparison accuracy.

Inventive Principle:

Principle #6Universality (Multi-functionality)

4Productivity

If automated data validation removes unnecessary data before comparison, then processing efficiency is improved, but data integrity risks increase

Engineering Contradiction:

Improveprocessing efficiencyVSAvoiddata integrity

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent extracts only the attributes specified in the mapping file for comparison while leaving the original files intact. This extraction approach improves processing efficiency by focusing on relevant data while maintaining data integrity through the use of read-only operations and unchanged source files.

Inventive Principle:

Principle #2Taking out (Extraction)

Data Source

PatentUS9053112B2Automated data validation

Publication Date: 2015.06.09 BANK OF AMERICA CORP

AI summary

According to some embodiments, logic executing on a processor receives a request to compare a first file and a second file. Each file comprises records, attributes, and attribute values. An attribute value is a value that a record associates with a corresponding attribute. The logic receives a mapping file indicating a key and one or more selected attributes for comparison. The logic compares each record in the first file to its corresponding record in the second file, the corresponding record determined according to the key. For records that fail to match, the logic determines which of the selected attributes are unmatched. The logic communicates a report indicating a result of comparing the first file and the second file.