Data Quality Metrics Analysis with Dynamic Weighting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data quality analysis techniques lack efficiency in allocating and utilizing computing resources due to static data quality metrics that do not account for constantly changing new or updated data over time.
Innovation Solution
A computer-implemented method that dynamically updates data quality metrics by tracking changes over time, assigning weights based on recency, and identifying anomalous data points, while efficiently managing computational resources through prioritization and incremental computations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data quality metrics are updated continuously with new data, then data quality assessment accuracy is improved, but computational resource consumption increases
Solution Approach 1:
The patent applies partial action by selectively updating only those data quality metrics that are affected by incoming data changes, rather than recomputing all metrics. The system identifies which metrics require updates based on the nature of new data arrivals, performing computations only where necessary to maintain assessment accuracy while reducing overall computational overhead.
Solution Approach 2:
The patent implements dynamics by making the data quality metric update process adaptive and flexible. The system dynamically determines which metrics to update based on incoming data characteristics, adjusting the update frequency and scope in real-time. This dynamic approach allows the system to maintain high assessment accuracy while optimizing computational resource usage according to actual data conditions.
2Productivity
If incremental computation is used for data quality metrics, then computational efficiency is improved, but metric accuracy may deteriorate due to static nature of incremental updates
Solution Approach 1:
The patent applies feedback by continuously monitoring the quality and recency of incremental metric updates. The system uses feedback signals from incoming data to determine when incremental computations are sufficient and when full recomputations are necessary to maintain accuracy. This feedback mechanism ensures that incremental computation efficiency is maintained while preventing accuracy degradation through timely full updates.
Solution Approach 2:
The patent implements continuity of useful action by maintaining an ongoing incremental computation process that continuously processes incoming data. Rather than periodic batch updates, the system performs continuous incremental computations that steadily improve metric accuracy over time while maintaining high computational efficiency, ensuring the useful action of data quality assessment continues without interruption.
3Reliability
If all data quality metrics are computed and updated, then comprehensive data quality assessment is achieved, but computational overhead increases
Solution Approach 1:
The patent applies local quality by differentiating the update frequency and depth for different data quality metrics based on their specific characteristics and importance. Rather than uniformly updating all metrics with the same computational intensity, the system applies localized computational resources to each metric according to its needs, achieving comprehensive assessment while minimizing overall computational overhead through selective, targeted updates.
Data Source
AI summary
Embodiments of the invention are directed a computer-implemented method for efficiently assessing data quality metrics. A non-limiting example of the computer-implemented method includes receiving, using a processor, a plurality of updates to data points in a data stream. The processor is further used to provide a plurality of data quality metrics (DQMs), and to maintain information on how much the plurality of DQMs are changing over time. The processor also maintains information on computational overhead for the plurality of DQMs, and also updates data quality information based on the maintained information.