Data Integrity Checks for Correctly Formatted Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems struggle to effectively detect and correct inconsistencies and errors in source data, particularly those that are correctly formatted, leading to inefficiencies in computational tools and a difficult, time-consuming process for creating custom monitoring tools.
Innovation Solution
A data integrity check system that performs agnostic data integrity checks using Boolean checks and generates human-readable reports to identify errors and anomalies in source data, incorporating historical data comparisons and user-defined thresholds for anomaly detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If automated systems monitor data streams to detect errors, then detection capability is improved, but the systems cannot detect incorrectly formatted values that are correctly formatted
Solution Approach 1:
The patent segments error detection into multiple independent analysis layers: format validation, statistical anomaly detection, and historical comparison. Each layer handles specific types of errors independently, allowing the system to detect both obviously formatted errors and subtle correctly-formatted anomalies without conflating detection methods.
Solution Approach 2:
The patent introduces an intermediary analysis layer that sits between raw data ingestion and final error identification. This intermediary layer performs statistical analysis and historical comparison on correctly-formatted data, acting as a mediator that detects anomalies without requiring format validation errors, thus completing the detection capability.
2Measurement precision
If custom tools are created to monitor specific data streams, then detection effectiveness is improved, but the process becomes difficult and time consuming
Solution Approach 1:
The patent creates a universal monitoring system that can handle multiple data stream types and error patterns through a single platform. The system performs format validation, statistical analysis, and historical comparison across diverse data sources without requiring custom tool development, achieving both effectiveness and efficiency through multi-functionality.
Solution Approach 2:
The system automatically configures monitoring parameters and adapts to different data streams without requiring manual custom tool creation. It self-adjusts statistical thresholds and comparison parameters based on the specific data being monitored, eliminating the time-consuming custom tool development process while maintaining detection effectiveness.
3Quantity of substance
If data is retrieved from multiple sources, then data quantity is improved, but data quality deteriorates due to formatting errors and human error
Solution Approach 1:
The patent performs preliminary format validation and anomaly detection on data immediately upon ingestion from multiple sources, before the data enters the processing pipeline. This preliminary action identifies and flags formatting errors and anomalies early, preventing propagation of low-quality data through the system and maintaining overall data quality despite high data quantity intake.
Data Source
AI summary
Aspects of the present disclosure relate to performing agnostic data integrity checks on source data, and based on the data integrity checks, generating a human-readable report that may be useable to identify specific errors or anomalies within the source data. Example embodiments involve systems and methods for performing the data integrity checks and generating the human-readable reports. For example, the method may include operations to ingest data from a source database through a data pipeline and into a local database, access the data from the data pipeline, determine a data type of the data, determine subtypes of data elements which make up the data, determine a count of each subtype, and generate a human-readable report, to be displayed at a client device.