Automated Data Quality Verification System

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Existing data quality validation techniques are inefficient and labor-intensive, particularly when dealing with large volumes of time-sensitive data, leading to delays and inaccuracies in data analysis.

Innovation Solution

An automated data quality assurance process that utilizes a system comprising multiple computing apparatuses to periodically update and maintain source and batch data tables, identifying new, changed, or expired records, and applying defined parameters and rule-based logic thresholds to ensure data accuracy and timeliness.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If manual data validation techniques are used, then data quality can be maintained, but the process becomes labor-intensive and inefficient

Engineering Contradiction:
Improvedata validation efficiencyVSAvoidtime for data analysis
Core Design Contradiction:
ProductivityVSLoss of time

Solution Approach 1:

The system implements automated self-validation mechanisms where the data processing system automatically validates its own output data against predefined rules, thresholds, and quality standards without requiring manual intervention. The system autonomously identifies data quality issues, generates validation reports, and triggers reprocessing when necessary, enabling the system to service itself and eliminating dependence on manual validation operations.

Inventive Principle:
Principle #25Self-service

Solution Approach 2:

The patent replaces manual mechanical validation processes with automated electronic validation mechanisms. The system uses computer-based algorithms, automated rule engines, and electronic data processing to substitute human operators in the validation process, thereby increasing efficiency and reducing the time required for data validation while maintaining or improving validation quality.

Inventive Principle:
Principle #28Mechanics substitution (Replace mechanical system)

2Quantity of substance

If data volume increases, then more comprehensive analysis is possible, but data quality maintenance becomes more difficult

Engineering Contradiction:
Improvedata volumeVSAvoiddata quality
Core Design Contradiction:
Quantity of substanceVSReliability

Solution Approach 1:

The system segments the validation process into distinct modular components including rule-based validation, threshold-based validation, and quality metric calculation. Each segment handles specific aspects of data validation independently, allowing the system to process large volumes of data through multiple specialized validation stages, thereby maintaining data quality across increasing data volumes without overwhelming a single validation mechanism.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The patent implements a universal validation framework that can handle multiple data types, formats, and quality standards through a single multi-functional system. The validation engine is designed to accommodate various validation rules, thresholds, and quality metrics applicable to different datasets, enabling consistent quality maintenance across diverse and expanding data volumes without requiring separate validation systems for each data type.

Inventive Principle:
Principle #6Universality (Multi-functionality)

3Productivity

If automated validation is implemented, then processing speed increases, but system complexity increases

Engineering Contradiction:
Improvevalidation processing speedVSAvoidsystem complexity
Core Design Contradiction:
ProductivityVSDevice complexity

Solution Approach 1:

The system manages complexity by dynamically adjusting validation parameters such as rule thresholds, sampling rates, and validation depth based on data characteristics and quality requirements. The automated system can modify these parameters in response to changing data conditions, allowing flexible adaptation without requiring complex fixed validation logic, thereby maintaining processing speed while managing system complexity through adaptive parameter management rather than rigid complex structures.

Inventive Principle:
Principle #35Parameter changes

Data Source

PatentUS20250110945A1System, apparatus, and method for maintaining data quality using automatic timeliness verification mechanisms
Publication Date: 2025.04.03 MORGAN STANLEY SERVICES GROUP INC
  • US20250110945A1 patent drawing
  • US20250110945A1 patent drawing
  • US20250110945A1 patent drawing

AI summary

A system and method for maintaining data quality, including: periodically updating, at a first computing apparatus, a source data table maintained at a second computing apparatus at a first time interval; periodically updating, in an iterative or recursive manner at the first computing apparatus, a batch data table maintained at the second computing apparatus at a second time interval; obtaining, at the second computing apparatus from an information system, one or more data tables, metadata, and one or more transaction tables; determining, at the second computing apparatus, whether a refresh timestamp or a modification timestamp is within a predetermined time period from a timestamp of the one or more transaction tables; when it is within the predetermined time period, updating a data visualization application for reviewing the timestamp negative result, wherein the data visualization application comprises a graphical user interface that is accessible using a third computing apparatus.