Anomaly Detection in Prediction Models via Data Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing model learning methods face reduced accuracy due to anomalous instances in learning data, which are difficult to detect and remove without prior knowledge, affecting the prediction model's performance.
Innovation Solution
An information processing apparatus that calculates differences between objective and predicted values, generates division conditions for explanatory variables, and selects cleansing conditions to divide data into sections, identifying and excluding anomalous instances based on error scores and anomaly conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If outlier detection with reference to objective variable distribution is performed, then some anomalous instances can be removed, but anomalous instances without objective variable outliers cannot be detected and previous knowledge is required
Solution Approach 1:
The patent segments the learning data into multiple groups based on explanatory variables, and performs anomaly detection within each segment. This allows detection of anomalous instances that differ from local patterns even when they don't stand out in the overall distribution, thereby improving detection accuracy without requiring prior knowledge about specific anomalies.
Solution Approach 2:
The patent enables the data cleaning process to be self-service by automatically generating cleansing conditions from the data itself through grouping and comparison operations. The system identifies anomalous instances by comparing data patterns across groups without needing external previous knowledge, making the process adaptable to any dataset.
2Reliability
If cleansing conditions are set based on previous knowledge, then outlier removal can be performed, but the process requires prior knowledge and cannot detect non-outlier anomalies
Solution Approach 1:
The patent performs preliminary grouping of data by explanatory variables before model training. This preliminary segmentation action reveals hidden patterns and anomalies that would be invisible in the raw data, enabling automatic detection of anomalous instances that affect model reliability without requiring complex prior knowledge.
Solution Approach 2:
The patent changes the parameter perspective from looking at objective variable distribution alone to examining relationships between explanatory variables and objective variables through grouping. This parameter transformation enables detection of anomalous instances based on their deviation from group patterns rather than requiring predefined knowledge about what constitutes an anomaly.
3Productivity
If all learning data is used for model generation, then data utilization is maximized, but anomalous instances reduce model accuracy
Solution Approach 1:
The patent extracts and identifies anomalous instances through grouping and comparison operations, then removes only these problematic data points from the learning set. This selective extraction approach maintains high data utilization efficiency by keeping most valid data while eliminating the small portion that would harm model accuracy.
Solution Approach 2:
The patent implements a feedback mechanism where the model training process itself provides information about data quality. By comparing predicted values with actual values within groups, the system identifies anomalous instances and feeds this information back to refine the learning dataset, thereby improving model accuracy while maintaining high data utilization.
Data Source
AI summary
According to one embodiment, an information processing apparatus includes: a difference calculator configured to calculate, based on a plurality of data items each including an objective variable and at least one explanatory variable, a plurality of differences between the objective variable and prediction values of the objective variable predicted from the explanatory variable; a candidate generator configured to generate at least one division condition candidate which divides the explanatory variable into a plurality of sections; and a division condition selector configured to select a first division condition from the at least one division condition candidate, based on the differences calculated by the difference calculator.