Ground Truth Dataset for Data Correctness Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for determining data correctness, such as On Target Reach (OTR) tests, are expensive, limited by small sample sizes, and prone to significant sampling errors, making it costly and inefficient to ensure data correctness across multiple data sources.
Innovation Solution
A computer-implemented method using a ground truth dataset to determine data correctness values by receiving datasets from various sources, associating each attribute value with a single correctness value, and creating probability distributions for these values, allowing for reduced costs and improved accuracy in data correctness management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If panel-based OTR tests are used to determine data correctness, then data correctness can be assessed, but the number of data points is limited and sampling error is significant
Solution Approach 1:
The patent creates a synthetic ground truth dataset by copying and transforming existing data elements. Each data element is replicated multiple times with different correctness values assigned according to probability distributions, thereby expanding the quantity of data points while maintaining representativeness of the original data characteristics.
Solution Approach 2:
The patent performs preliminary actions by pre-calculating and assigning correctness values to data elements before they are used for assessment. Correctness values are determined in advance based on probability distributions and stored in the ground truth dataset, eliminating the need for time-consuming external verification during actual data correctness assessment.
2Reliability
If external correctness checks are performed for each data subset, then data correctness can be verified, but costs increase significantly
Solution Approach 1:
The patent performs preliminary action by pre-determining and storing correctness values for data elements in the ground truth dataset before they are needed for verification. This eliminates the need for expensive external correctness checks during actual verification processes, as the correctness information is already available from the pre-computed ground truth dataset.
Solution Approach 2:
The patent enables self-service by allowing the ground truth dataset to provide its own correctness verification capability. The dataset contains embedded correctness values that can be directly queried and used for verification without requiring external verification services, thereby reducing costs while maintaining reliability.
3Measurement precision
If panel-based OTR tests are used, then data correctness can be measured, but sampling error is significantly large
Solution Approach 1:
The patent reduces sampling error by copying and replicating data elements multiple times in the ground truth dataset. Each data element is replicated with appropriate correctness values assigned according to probability distributions, creating a larger and more representative sample that accurately reflects the underlying data characteristics without the significant sampling errors associated with limited panel-based samples.
Data Source
AI summary
Method and system for providing ground truth dataset and use thereof for improving data correctness. Datasets comprising data elements are received from different sources, each data element including an identifier and at least one attribute value associated therewith. Data correctness values are determined for the attribute values, each associated with a probability that an attribute value is correct. Data element with single data correctness value is added to the ground truth dataset for each attribute value for each identifier with which a respective attribute value is associated based on the determined data correctness values for the attribute values, whereby the data correctness values in the ground truth dataset define probability distributions of data correctness for the attribute values. Data correctness values for attribute values of data elements of a new dataset can be determined based on overlapping data compared in the ground truth dataset and the new dataset.