Data Quality Rule Scheduling for Large-Scale Error Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The exponential growth of data collection has led to the acquisition of low quality data, including technical issues and incorrect business logic, which complicates data analysis and management.
Innovation Solution
A computer system and method for managing data quality rules, involving a server device and database that schedule, execute, and track data quality rules, categorize them, and generate trend reports to ensure data consistency and adherence to standards, using modules like rule scheduler, data quality rule module, and rule versioning module.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If large amounts of data are collected to improve analysis coverage, then data quantity increases, but data quality deteriorates due to capturing low quality data
Solution Approach 1:
The system executes data quality rules before data is stored or before analysis is performed. The rule scheduler triggers quality checks in advance, and the data quality rule module validates data against predefined criteria before it enters the database, preventing low quality data from being stored while maintaining the ability to collect large amounts of data.
Solution Approach 2:
The system implements feedback mechanisms where the data quality rule module receives output from rule execution and stores this output in the database. The rule scheduler can re-execute rules based on previous outputs, creating a feedback loop that continuously monitors and improves data quality while maintaining comprehensive data collection.
2Reliability
If data quality rules are executed frequently to improve data quality, then reliability improves, but system complexity increases
Solution Approach 1:
The system segments data quality rules into distinct categories (e.g., syntax rules, business logic rules, format rules). The rule scheduler manages these categories separately and can execute them independently based on triggers. This segmentation reduces overall system complexity by allowing modular management and execution of quality checks.
Solution Approach 2:
The data quality rule module acts as an intermediary between the data collection system and the analysis system. It receives data, executes rules, and returns validated data or error outputs. This intermediary layer simplifies the overall system architecture by centralizing quality check logic in a dedicated module that can be independently managed and executed.
3Difficulty of detecting and measuring
If multiple data quality rules are executed to detect all types of errors, then detection capability improves, but execution time increases
Solution Approach 1:
The rule scheduler executes data quality rules periodically based on triggers rather than continuously. Rules are executed at specific intervals or when certain conditions are met, reducing total execution time while maintaining comprehensive error detection capability. The scheduler can adjust execution frequency based on data volume and criticality.
Solution Approach 2:
The system applies different quality check intensities to different data categories. Critical data fields undergo more rigorous checking with multiple rules, while less critical fields undergo lighter checking. This local quality approach ensures comprehensive error detection where needed while reducing overall execution time by applying appropriate checking intensity to each data type.
Data Source
AI summary
An example computer system for executing data quality rules, the computer system comprising one or more processors; and non-transitory computer-readable storage media encoding instructions which, when executed by the one or more processors, causes the computer system to: receive a plurality of rules; group the plurality of rules into one or more categories of rules; determine a category of the one or more categories of rules to execute based on a scheduling trigger; request execution of each rule of the category by a database; and receive, from the database, output from execution of the each rule of the category.