Generalized Linear Model Selection Framework for Time Series Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Choosing the appropriate Generalized Linear Model (GLM) for count data is challenging due to numerous available models and inadequate documentation, particularly in determining overdispersion or underdispersion, which affects anomaly detection in time series data.
Innovation Solution
A framework that includes a dispersion determination module to identify whether data is overdispersed or underdispersed, and a GLM selector module to automatically choose the appropriate GLM based on this determination, using modules such as Poisson, negative binomial, zero-inflated, and hurdle models, to apply on time series count data for anomaly detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple GLM options are provided to handle different data characteristics, then the model's adaptability to various count data types improves, but the complexity of model selection increases
Solution Approach 1:
The system automatically changes model parameters by examining data characteristics (overdispersion, underdispersion, zero-inflation) and selecting appropriate GLM variants. This resolves the contradiction by allowing multiple specialized models to handle different data types while the automated parameter selection process prevents user-facing complexity.
Solution Approach 2:
The framework performs self-service by automatically determining the appropriate GLM model through statistical tests on the input data. The system evaluates dispersion patterns and zero-inflation levels, then autonomously selects the most suitable model without requiring user expertise in model selection criteria.
2Ease of operation
If automated model selection is implemented, then ease of operation improves, but the need for statistical analysis increases computational complexity
Solution Approach 1:
The system performs preliminary statistical analysis (calculating dispersion parameters, testing for zero-inflation) before model selection. This preliminary action automates the decision-making process, improving ease of operation while managing computational complexity through efficient statistical tests rather than exhaustive model comparisons.
Solution Approach 2:
The framework replaces manual statistical analysis mechanics with automated computational procedures. Statistical tests for overdispersion, underdispersion, and zero-inflation are automatically executed, substituting the need for user-performed mechanical statistical work with streamlined computational processes.
3Loss of information
If comprehensive documentation is provided for all GLM options, then information completeness improves, but the difficulty of understanding increases for users without statistical background
Solution Approach 1:
The framework extracts and applies only the necessary statistical information from comprehensive documentation, filtering out unnecessary complexity. It automatically determines model parameters based on data characteristics without requiring users to understand or navigate complex statistical theory, thus maintaining information completeness while reducing understanding difficulty.
Solution Approach 2:
The automated model selection process acts as an intermediary between comprehensive statistical documentation and end users. It translates complex statistical concepts into automatic decisions based on simple data characteristics, preserving the completeness of statistical information while shielding users from complexity through automated interpretation.
Data Source
AI summary
Systems and methods are provided framework for automatically choosing the appropriate generalized linear model (GLM) given a time series of count data, and for anomaly detection on time series data. A dispersion parameter is determined and used to determine whether the count data is overdispersed data or underdispersed data. The overdispersed data or the underdispersed data is used to determine a GLM to apply on the dataset. Using the determined GLM on the data, anomalies can be determined.