Dynamic Data Quality Assessment Using Change Data Thresholds
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data quality assessment methods consume excessive computing resources and compromise system security by requiring frequent downloads of large data tables, and lack a standardized definition of data quality, hindering efficient digital transformation research and development.
Innovation Solution
A data processing device and method that calculates and tracks data quality assessment measurement values dynamically, using a data quality assessment measurement calculation module to evaluate changes in data tables without reacquiring the entire data set, and sends notifications when quality meets predefined thresholds.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data users download all data in huge data tables to local computers every day for calculating data quality, then data quality assessment accuracy is improved, but computing resource consumption increases and system information security is compromised
Solution Approach 1:
The patent extracts only the necessary data quality assessment calculations from the complete data set by implementing a server-side calculation mechanism. The server computes data quality metrics directly on stored data without requiring full data downloads, thereby maintaining assessment accuracy while eliminating excessive computing resource consumption at user endpoints.
Solution Approach 2:
The patent introduces a server as an intermediary between the data storage system and user devices. This server performs data quality assessment calculations centrally, acting as a mediator that provides accurate quality metrics to users without requiring them to handle the actual large data sets, thus resolving the contradiction between accuracy and resource consumption.
2Measurement precision
If data users download all data in huge data tables to local computers every day for calculating data quality, then data quality assessment accuracy is improved, but system information security is compromised
Solution Approach 1:
The patent extracts the data quality assessment function from user local computers and relocates it to a secure server environment. This extraction eliminates the need for users to download sensitive data to their local systems, maintaining assessment accuracy while removing the security vulnerability of data exposure during transfer and storage at user endpoints.
Solution Approach 2:
The server acts as a secure intermediary that handles all data quality calculations within the protected data center environment. Users interact only with the server through controlled interfaces, preventing direct access to raw data and thereby eliminating the security risks associated with downloading and storing large data tables locally.
3Measurement precision
If data quality assessment is performed by downloading complete data tables, then assessment completeness is improved, but time consumption increases
Solution Approach 1:
The patent implements preliminary action by pre-computing and maintaining data quality assessment metrics on the server side as data is updated. When new data arrives or existing data changes, the server automatically recalculates quality metrics and stores them ready for immediate retrieval, eliminating the need for users to perform time-consuming downloads and local calculations.
Solution Approach 2:
The patent establishes continuous data quality assessment by implementing automatic, ongoing calculations on the server whenever data changes occur. This continuous process ensures assessment completeness is maintained without interruption while eliminating periodic time-consuming batch downloads, as the quality metrics are continuously updated and immediately available to users.
Data Source
AI summary
A data processing device is provided, which includes a storage device and a data quality assessment measurement calculation module. The storage device is configured to obtain change data of data tables from a data center. The data quality assessment measurement calculation module is configured to calculate an updated data quality assessment measurement value and an updated data quality characteristic data according to the change data and a data quality assessment measurement reference value and perform a notification function according to the updated data quality assessment measurement value and a data quality threshold value. The data quality assessment measurement calculation module is configured to compare the updated data quality assessment measurement value with the data quality threshold value to generate a comparison result. The data quality assessment measurement calculation module is configured to generate and send a notification signal to perform a notification function according to the comparison result.