Real-time Data Quality Analysis with Adaptive Weighting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data quality analysis techniques lack efficiency in allocating and utilizing computing resources due to static data quality metrics that do not account for the constant change of new or updated data over time, leading to suboptimal performance in identifying and addressing data quality issues in big data sets.
Innovation Solution
A computer-implemented method that computes data quality metrics (DQMs) by differentiating data points over time, assigning higher weights to more recent data points, and continuously updating them while limiting processor cycles, along with identifying and assigning lower weights to anomalous data points based on their severity, ensuring efficient resource allocation and data quality assessment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data quality metrics are continuously updated on constantly changing data streams, then data quality analysis accuracy is improved, but computing resource consumption increases
Solution Approach 1:
The patent applies dynamics by making the data quality metric computation adaptive to changing data characteristics. The system dynamically adjusts which metrics to compute and how frequently based on detected changes in data distribution, ensuring accurate quality assessment while avoiding unnecessary computations on stable data portions.
Solution Approach 2:
The system changes parameters by using different computation frequencies and depths for different data quality metrics based on their importance and the observed data stability. High-priority metrics are updated more frequently while low-priority metrics use coarser update intervals, optimizing the balance between accuracy and resource usage.
2Productivity
If incremental computation of data quality metrics is used for new/updated data, then computing resource efficiency is improved, but the static nature of metrics fails to capture constant data changes over time
Solution Approach 1:
The system implements feedback mechanisms that monitor data distribution changes and use this information to adjust metric computation strategies. When significant changes are detected in data patterns, the system increases computation frequency and depth for affected metrics, ensuring adaptability while maintaining efficiency during stable periods.
Solution Approach 2:
The patent employs periodic action by scheduling metric computations at variable intervals based on data stability. Instead of uniform periodic updates, the system adjusts computation timing dynamically - using longer intervals when data is stable and shorter intervals when changes are detected, thus balancing efficiency and responsiveness.
3Reliability
If comprehensive data quality checks are performed on entire data sets, then data quality issue detection is improved, but processing time increases
Solution Approach 1:
The system segments the data stream into manageable portions and applies different computation strategies to different segments. High-priority quality checks are applied to all segments, while lower-priority checks are applied selectively based on segment characteristics and change detection, reducing overall processing time while maintaining detection reliability.
Solution Approach 2:
The patent applies partial action by performing comprehensive data quality checks only when necessary - such as when significant changes are detected or when high-stakes decisions are being made. During normal operation, a subset of critical metrics is monitored continuously at a reduced computation level, providing sufficient quality assurance with minimal processing time.
Data Source
AI summary
Embodiments of the invention are directed a computer-implemented method for assessing data quality. A non-limiting example of the computer-implemented method includes using a processor to receive a plurality of updates to data points in a data stream. The processor is further used to compute instances of a data quality metric (DQM) from the data points in the data stream. The instances of the DQM are configured to differentiate the data points in the data stream by time and assign a higher weight to the instances of the DQM computed from more recent data points in the data stream. The instances of the DQM are continuously updated as more of the data points are received by the processor while limiting cycles of the processor consumed by updating the instances of the DQM to a threshold.