Continuous Data Quality System for Defect Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data management systems face significant downtime when correcting historical data defects, as they require manual deletion and reloading of data, which can take weeks or months, disrupting user access.
Innovation Solution
A continuous data quality system that automatically detects defects by comparing data from a source repository with a target repository, using a data manager and staging repositories to transform and update data, allowing for near-instant correction and reduction of downtime.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If manual deletion and reloading of data is performed to correct historical data defects, then data accuracy is improved, but system downtime increases significantly
Solution Approach 1:
The patent segments the data loading process into multiple parallel streams: (1) loading historical data portions from source repository through ETL processes, (2) loading target data portions directly from source repository, and (3) comparing the two sets of data. This segmentation allows defect detection and correction to occur without requiring complete system downtime, as each segment can be processed independently and concurrently with system operation.
Solution Approach 2:
The patent performs preliminary actions by loading historical data portions into a staging repository and comparing them with target data portions before actual defect correction. This preliminary comparison identifies defects in advance, allowing the system to prepare correction strategies without interrupting ongoing operations, thus reducing the actual downtime required for correction.
2Reliability
If complete data reloading is performed to ensure data quality, then data reliability is improved, but productivity decreases due to extended downtime
Solution Approach 1:
The patent maintains continuity of useful action by enabling the system to continue operating while data quality improvements are being made. The parallel loading and comparison processes run concurrently with system operations, allowing data reliability to be enhanced without interrupting productivity. Users can access the system throughout the entire process, and corrections are applied minimally rather than requiring complete system shutdown.
Solution Approach 2:
The patent introduces a staging repository as an intermediary between the source repository and the target data storage. This intermediary allows historical data to be loaded, transformed, and compared without directly impacting the operational target repository. The staging area serves as a buffer that enables data quality verification and correction preparation while the system remains operational, thus maintaining productivity while improving reliability.
3Productivity
If incremental data loading is used to maintain system availability, then system availability is improved, but defect detection capability is reduced
Solution Approach 1:
The patent segments data into historical data portions and target data portions, allowing comparative analysis even within an incremental loading framework. By dividing the data into comparable segments (historical vs. target portions with matching keysets), the system maintains defect detection capability while proceeding with incremental loading that preserves system availability.
Solution Approach 2:
The patent implements feedback mechanisms by comparing historical data portions with target data portions and identifying defects through systematic difference analysis. This feedback loop enables continuous defect detection during incremental loading operations, allowing the system to maintain both high availability and strong defect detection capability through automated comparison and identification processes.
Data Source
AI summary
The embodiments provide a system and method for continuously updating a target repository to include both the latest data and corrected historical data. The system includes a data manager and at least two staging repositories. Each time the system retrieves the latest data from the source repository it also retrieves a portion historical data. Both the latest data and the historical data are transformed, and the historical portion of the transformed data is compared with corresponding data from the target repository to determine if there are any defects in the data from the target repository. The target repository is automatically updated if a defect is detected.