Automated Data Quality Verification System
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data quality validation techniques are inefficient and labor-intensive, particularly when dealing with large volumes of time-sensitive data, leading to delays and inaccuracies in data analysis.
Innovation Solution
An automated data quality assurance process that utilizes a system comprising multiple computing apparatuses to periodically update and maintain source and batch data tables, identifying new, changed, or expired records, and applying defined parameters and rule-based logic thresholds to ensure data accuracy and timeliness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual data validation techniques are used, then data quality can be maintained, but the process becomes labor-intensive and inefficient
Solution Approach 1:
The system implements automated self-validation mechanisms where the data processing system automatically validates its own output data against predefined rules, thresholds, and quality standards without requiring manual intervention. The system autonomously identifies data quality issues, generates validation reports, and triggers reprocessing when necessary, enabling the system to service itself and eliminating dependence on manual validation operations.
Solution Approach 2:
The patent replaces manual mechanical validation processes with automated electronic validation mechanisms. The system uses computer-based algorithms, automated rule engines, and electronic data processing to substitute human operators in the validation process, thereby increasing efficiency and reducing the time required for data validation while maintaining or improving validation quality.
2Quantity of substance
If data volume increases, then more comprehensive analysis is possible, but data quality maintenance becomes more difficult
Solution Approach 1:
The system segments the validation process into distinct modular components including rule-based validation, threshold-based validation, and quality metric calculation. Each segment handles specific aspects of data validation independently, allowing the system to process large volumes of data through multiple specialized validation stages, thereby maintaining data quality across increasing data volumes without overwhelming a single validation mechanism.
Solution Approach 2:
The patent implements a universal validation framework that can handle multiple data types, formats, and quality standards through a single multi-functional system. The validation engine is designed to accommodate various validation rules, thresholds, and quality metrics applicable to different datasets, enabling consistent quality maintenance across diverse and expanding data volumes without requiring separate validation systems for each data type.
3Productivity
If automated validation is implemented, then processing speed increases, but system complexity increases
Solution Approach 1:
The system manages complexity by dynamically adjusting validation parameters such as rule thresholds, sampling rates, and validation depth based on data characteristics and quality requirements. The automated system can modify these parameters in response to changing data conditions, allowing flexible adaptation without requiring complex fixed validation logic, thereby maintaining processing speed while managing system complexity through adaptive parameter management rather than rigid complex structures.
Data Source
AI summary
A system and method for maintaining data quality, including: periodically updating, at a first computing apparatus, a source data table maintained at a second computing apparatus at a first time interval; periodically updating, in an iterative or recursive manner at the first computing apparatus, a batch data table maintained at the second computing apparatus at a second time interval; obtaining, at the second computing apparatus from an information system, one or more data tables, metadata, and one or more transaction tables; determining, at the second computing apparatus, whether a refresh timestamp or a modification timestamp is within a predetermined time period from a timestamp of the one or more transaction tables; when it is within the predetermined time period, updating a data visualization application for reviewing the timestamp negative result, wherein the data visualization application comprises a graphical user interface that is accessible using a third computing apparatus.


