Automated Bulk Data Conversion Validation Framework

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data conversion systems face challenges in accurately validating large datasets for downstream systems, leading to potential defects and program failures, necessitating a solution for comprehensive validation, segregation of data types by criticality, and efficient data lifecycle management.

Innovation Solution

A system for automating bulk data conversion that involves generating a relational database conversion template, creating a peer review data package, and uploading it for access by multiple users, with features like shadow data comparison and validation results packaging for downstream review and certification.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If comprehensive validation of large datasets is performed manually, then validation accuracy is improved, but time consumption and resource requirements increase significantly

Engineering Contradiction:

Improvevalidation accuracyVSAvoidtime consumption

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system enables self-service validation through automated execution of validation rules and procedures against converted data. The framework automatically compares source data with converted data, executes validation logic, and generates validation reports without requiring manual intervention, thereby maintaining high validation accuracy while dramatically reducing time consumption.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent replaces manual mechanical validation processes with automated computational systems. Validation rules, procedures, and comparisons are executed through software frameworks that automatically process large datasets, substituting human manual review with machine-based validation that achieves both high accuracy and efficiency.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Reliability

If detailed review features and comprehensive controls are implemented, then data conversion reliability is improved, but system complexity increases

Engineering Contradiction:

Improvedata conversion reliabilityVSAvoidsystem complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The validation framework is segmented into modular components including validation rules, validation procedures, validation configurations, and execution engines. Each module handles specific aspects of validation independently, allowing comprehensive controls to be implemented through organized, manageable segments rather than monolithic complex systems.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system manages complexity through parameterized validation rules and configurable validation procedures. By allowing validation criteria to be defined as adjustable parameters rather than hard-coded logic, the system can adapt to different data conversion scenarios while maintaining a consistent underlying framework, thereby improving reliability without proportionally increasing complexity.

Inventive Principle:

Principle #35Parameter changes

3Loss of information

If legacy data and converted data are archived for future review, then traceability and auditability are improved, but storage requirements and data management complexity increase

Engineering Contradiction:

ImprovetraceabilityVSAvoidstorage requirements

Core Design Contradiction:

Loss of informationVSQuantity of substance

Solution Approach 1:

The system extracts and archives only the essential validation information and critical data subsets rather than storing complete datasets. Validation results, comparison outcomes, and key metadata are preserved for traceability, while full raw data archives are minimized, thereby maintaining auditability while reducing storage requirements.

Inventive Principle:

Principle #2Taking out (Extraction)

4Adaptability or versatility

If the validation framework is made modular and adaptable for future upgradability, then system flexibility is improved, but initial development complexity increases

Engineering Contradiction:

Improvesystem flexibilityVSAvoiddevelopment complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The validation framework is designed with universal, multi-functional components that can handle various data conversion scenarios through a single unified system. The modular architecture allows the same core validation engine to adapt to different validation rules, procedures, and data types, providing future upgradability and flexibility without requiring separate development for each scenario.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS11461357B2Data conversion bulk validation automated framework

Publication Date: 2022.10.04 BANK OF AMERICA CORP

AI summary

Embodiments of the invention are directed to systems, methods, and computer program products for automating bulk data conversion processes of one or more database management systems. Data conversion projects of focus may comprise conversion of a large bulk of data with a wide range in order of magnitude. The system is designed and driven by the present constraints of large data conversion and is based on principles of reviewability, minimization of manual review and development work, persistence of data stores for data result comparison, process optimization for downstream review and certification, timely execution, and allowance for concurrent development by multiple systems and resources.