Anomalous Data Curation Through Source-Verified Pipeline Screening
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Insufficient data curation resources lead to uncurated or partially curated data, which can negatively impact the quality and availability of computer-implemented services, particularly in cases where anomalous data points with subjective features are difficult to verify.
Innovation Solution
Engage the data source to verify and correct anomalous data points by interacting with them through importance criteria comparison and feedback, reducing the reliance on limited data curation resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data curation resources are increased to verify all anomalous data points, then data quality and trustworthiness improve, but resource consumption and operational complexity increase
Solution Approach 1:
The patent applies local quality by differentiating verification approaches based on data point characteristics. Instead of uniformly verifying all anomalous data points with extensive resources, the system identifies and applies targeted verification methods specific to each anomaly type and data source capability, optimizing resource allocation while maintaining data trustworthiness
Solution Approach 2:
The patent implements self-service by enabling data sources to autonomously verify and correct their own anomalous data points through interactive processes. The data source system itself participates in the verification workflow, reducing the burden on external curation resources while improving data quality through source-validated corrections
2Measurement precision
If automated anomaly detection is applied to all data points, then anomaly detection precision improves, but processing time and computational resources increase
Solution Approach 1:
The patent applies segmentation by dividing the data processing workflow into distinct phases: initial automated screening of data points, identification of anomalous candidates, and subsequent interactive verification. This segmentation allows automated precision detection to focus only on potential anomalies rather than processing all data points equally, reducing overall processing time while maintaining detection precision
Solution Approach 2:
The patent implements partial action by applying automated anomaly detection selectively rather than exhaustively to all data points. The system performs automated detection on a subset of data or uses preliminary filtering to identify only those points requiring detailed analysis, thereby reducing computational overhead while maintaining precision for critical anomalies
3Manufacturing precision
If interactive verification with data sources is implemented for all anomalous points, then data accuracy improves, but processing throughput decreases
Solution Approach 1:
The patent applies local quality by implementing interactive verification selectively for specific types of anomalies or data sources based on their reliability characteristics. High-trust data sources may receive automated processing while lower-trust sources undergo interactive verification, maintaining data accuracy for critical sources while preserving overall processing throughput
Solution Approach 2:
The patent implements partial action by applying interactive verification to only a portion of anomalous data points rather than all of them. The system prioritizes verification for high-impact or high-uncertainty anomalies while using automated methods for lower-priority cases, thereby maintaining data accuracy for critical points while preserving processing throughput
Data Source
AI summary
Methods and systems for curating data by a data manager are disclosed. Data may be curated from various data sources before being provided to downstream consumers that may rely on the trustworthiness of the curated data in order to provide desired computer-implemented services. During the data curation process, data curation resources are used to improve the trustworthiness and/or value of the collected data. However, data curation resources (e.g., data curators, computing resources) may be limited and/or insufficient to perform the data curation process as desired, which may result in unusable and/or uncurated (e.g., untrustworthy) data. Thus, the data may be screened for anomalous data points. Features of the anomalous data points that meet importance criteria may be presented to the data source and the data source may indicate whether the features are expected. If the features are expected, the data pipeline may be populated with the anomalous data points.