Category-Based Data Quality Rule Execution with Trend Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The exponential growth of data collection has led to the acquisition of low quality data, including technical errors and incorrect business logic, which complicates data analysis and management.
Innovation Solution
A computer system and method for managing data quality rules, involving a server device and database that schedules, executes, and tracks data quality rules, categorizes them, and generates trend reports to ensure data consistency and adherence to standards, using modules for scheduling, roster management, versioning, and categorization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If large amounts of data are collected to improve analysis coverage, then data quantity increases, but data quality deteriorates due to capturing low quality data with technical errors and incorrect business logic
Solution Approach 1:
The system executes data quality rules before data is stored in the database, performing validation and correction actions in advance. This preliminary action filters out low quality data and corrects errors before they enter the storage system, thereby maintaining high data quality while allowing extensive data collection.
Solution Approach 2:
The system continuously monitors data quality by executing rules and receiving feedback about data accuracy, completeness, and consistency. This feedback loop enables the system to detect and correct low quality data, ensuring that data collected in large quantities maintains high reliability standards.
2Reliability
If data quality rules are executed frequently to improve data accuracy, then data quality improves, but system complexity and processing time increase
Solution Approach 1:
The system segments data quality rules into distinct categories (e.g., accuracy rules, completeness rules, consistency rules) and executes them separately. This segmentation allows the system to manage complexity by processing rules in organized groups rather than as a single complex monolithic process.
Solution Approach 2:
The system executes data quality rules periodically based on scheduling triggers rather than continuously. This periodic execution reduces system complexity and processing overhead while maintaining data quality by performing validations at appropriate intervals rather than constantly.
3Reliability
If multiple data quality rules are applied to ensure compliance, then data reliability improves, but processing time and operational complexity increase
Solution Approach 1:
The system performs data quality validations in advance before data storage, so that compliance checks are completed beforehand. This preliminary action reduces processing time during subsequent operations by ensuring data is already validated and compliant.
Solution Approach 2:
The system combines multiple data quality rules into unified categories and executes them as integrated processes. This merging approach reduces the number of separate processing steps and minimizes total processing time while maintaining comprehensive compliance coverage.
Data Source
AI summary
An example computer system for executing data quality rules, the computer system comprising one or more processors; and non-transitory computer-readable storage media encoding instructions which, when executed by the one or more processors, causes the computer system to: receive a plurality of rules; group the plurality of rules into one or more categories of rules; determine a category of the one or more categories of rules to execute based on a scheduling trigger; request execution of each rule of the category by a database; and receive, from the database, output from execution of the each rule of the category.