ML Data Load Screening to Block Erroneous Repository Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Faulty data introduced into database systems propagates and is difficult and costly to correct, affecting records, reports, and workflows due to incorrect time period associations.
Innovation Solution
A machine learning model classifies data loads as a whole, preventing storage of erroneous data in a target repository and generating notifications, trained using statistical relationships and previous data loads with similar characteristics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional data validation methods are used to check individual data points, then data quality can be maintained, but processing time and computational resources increase significantly
Solution Approach 1:
The system performs preliminary classification of entire data loads using a machine learning model before individual data points are processed or stored. By predicting whether a data load is erroneous at the aggregate level, the system avoids the time-consuming process of validating each individual data point while still maintaining data quality through early rejection of problematic data loads.
Solution Approach 2:
The invention extracts and analyzes aggregate-level features from data loads (such as time period characteristics, data volume patterns, and source metadata) separately from individual data point validation. This extraction of high-level characteristics enables rapid classification without requiring detailed examination of each data point, thus reducing processing time while maintaining reliability.
2Productivity
If faulty data is allowed to propagate through the database system, then processing speed is maintained, but data corruption spreads to records, reports, and workflows
Solution Approach 1:
The system applies preliminary anti-action by predicting and preventing erroneous data loads from entering the database system in the first place. The machine learning model analyzes incoming data loads and identifies potential errors before the data is processed or stored, thereby stopping data corruption at its source rather than attempting to detect and correct it after propagation has occurred.
Solution Approach 2:
By performing classification before data entry, the system takes preliminary action to prevent faulty data from propagating. This early detection mechanism maintains processing speed because good data flows through the system without interruption, while erroneous data is blocked before it can affect records, reports, or workflows.
3Measurement precision
If comprehensive data validation is performed on every data load, then data accuracy is improved, but system complexity and computational overhead increase
Solution Approach 1:
The system segments the data validation process into two distinct stages: (1) a lightweight machine learning-based classification stage that evaluates entire data loads at aggregate level, and (2) a selective detailed validation stage that is only applied to data loads that pass the initial classification. This segmentation reduces system complexity by avoiding comprehensive validation of all data while maintaining accuracy through targeted verification.
Solution Approach 2:
The invention changes the parameters of validation from individual data point level to aggregate data load level. By transforming the validation approach to work with high-level features such as time period characteristics, data volume metrics, and source metadata, the system achieves comparable accuracy with significantly reduced computational overhead and system complexity.
4Quantity of substance
If erroneous data loads are stored in the database, then data repository capacity is utilized, but correction costs and effort increase significantly later
Solution Approach 1:
The system performs preliminary classification of data loads before storage, identifying and rejecting erroneous data in advance. This preliminary action prevents faulty data from occupying database storage capacity, thereby eliminating the need for costly post-storage correction efforts while maintaining efficient use of storage resources for valid data only.
Solution Approach 2:
The machine learning model's ability to identify erroneous data loads converts what would be a harmful storage of bad data into a beneficial filtering mechanism. By leveraging the model's predictive capabilities, the system transforms potential data quality problems into an opportunity for early detection and prevention, saving significant correction costs while optimizing storage capacity utilization.
Data Source
AI summary
Techniques for data intake that prevent corruption of data repositories with faulty data are disclosed. A data load may include individual values that are erroneous and individual values that are non-erroneous. A system uses a machine learning (ML) model trained to classify the data load, as a whole, as erroneous or non-erroneous. In a data intake process, the system applies the ML model to the data load. In response to determining that the data load is erroneous, the system prevents the storage of the data load within a target data repository.