Missing Value Identification Using Rules Graph and Risk Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in efficiently identifying and prioritizing missing values in data objects, as well as determining the criticality of these missing values across various data types and storage conditions.
Innovation Solution
The proposed solution involves a computer-implemented method that uses a deterministic rules graph and a trained machine learning model to identify missing values in data objects. The method generates a risk score for each output indicator based on the missing values, compares this score to a threshold, and displays alerts for missing values that exceed the threshold.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional methods are used to identify missing values in data objects, then the process is simple, but the identification efficiency is low and criticality assessment is insufficient
Solution Approach 1:
The system segments the missing value identification process into multiple independent components: a deterministic rules graph for structural analysis, a machine learning model for risk assessment, and a prioritization mechanism. This segmentation allows each component to specialize in specific aspects, improving overall efficiency while maintaining manageable complexity through modular architecture.
Solution Approach 2:
The patent introduces intermediary components including a deterministic rules graph that mediates between raw data and machine learning models, and a risk score calculation layer that bridges missing value detection with prioritization. These intermediaries transform the complex task into manageable stages, improving identification efficiency without requiring direct complex interactions between all system elements.
2Measurement precision
If all missing values are identified without prioritization, then completeness is high, but the ability to determine criticality and allocate resources is reduced
Solution Approach 1:
The system applies local quality by assigning different risk scores and priorities to different missing values based on their specific characteristics, relationships to other data elements, and impact on data quality. Not all missing values are treated equally; instead, each missing value receives a customized assessment that reflects its local importance, enabling precise criticality assessment and efficient resource allocation.
Solution Approach 2:
The patent transforms the identification process by introducing risk scores as a new parameter that changes the perspective from simple presence/absence of missing values to a prioritized list based on calculated risk. This parameter change enables the system to distinguish between critical and non-critical missing values, improving measurement precision while reducing the effective time needed by focusing efforts on high-risk items.
3Quantity of substance
If data objects from multiple sources are accessed, then data comprehensiveness is improved, but the difficulty of accessing and identifying missing values increases
Solution Approach 1:
The deterministic rules graph serves as a universal framework that can handle data objects from multiple sources and types. It provides a standardized way of representing relationships between data elements, making the system adaptable to various data structures without requiring source-specific processing logic. This multi-functionality improves comprehensiveness while managing detection difficulty through standardization.
Solution Approach 2:
The system creates a standardized representation (copy) of data relationships through the deterministic rules graph, which simplifies the complex task of detecting missing values across diverse data sources. Instead of analyzing each data source individually, the system works with this standardized model, reducing detection difficulty while maintaining comprehensiveness across all data objects.
Data Source
AI summary
Disclosed are systems and methods for receiving one or more data objects associated with an entity, each of the one or more data objects including one or more input indicators and one or more output indicators based on the one or more input indicators and determining, using a deterministic rules graph that maps each of the one or more input indicators to a corresponding one of the one or more output indicators, at least one of a first missing value for an input indicator or a second missing value for an output indicator. A trained machine learning model is used to generate a risk score associated with each of the one or more output indicators based on the at least one of the first missing value or the second missing value. The risk score is compared to a predetermined threshold value, and an alert is generated based on the comparing.