Real-Time Anomaly Detection Using Precomputed Statistical Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data analysis systems are inefficient in providing real-time insights due to the need for complete re-analysis of incoming data, which takes significant time and is not suitable for immediate action, especially when detecting trends or anomalies.
Innovation Solution
A method and system that utilize a statistical model derived from previous data records to quickly identify anomalies by comparing new data items with expected values, updating the model in real-time, and storing insights with associated data records for rapid retrieval and forensic investigation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If complete re-analysis of incoming data is performed, then identification of trends and anomalies is achieved, but processing time becomes too long for real-time requirements
Solution Approach 1:
The patent pre-calculates and stores statistical parameters (mean, standard deviation, minimum, maximum values) from historical data before real-time analysis is needed. When new data arrives, the system only needs to compare against these pre-computed statistics rather than re-analyzing all historical data, enabling real-time anomaly detection while maintaining accuracy
Solution Approach 2:
The patent divides the data analysis process into separate functional components: data ingestion, statistical parameter calculation, anomaly detection, and result storage. This segmentation allows each component to be optimized independently and enables parallel processing, reducing overall processing time while maintaining comprehensive analysis capabilities
2Adaptability or versatility
If statistical models are updated continuously with new data, then real-time adaptability is improved, but computational complexity increases
Solution Approach 1:
The patent updates only the necessary statistical parameters (mean, standard deviation, min, max) when new data arrives, rather than re-computing the entire statistical model from scratch. This partial update approach maintains model adaptability to new data patterns while significantly reducing computational complexity compared to full model re-training
Solution Approach 2:
The patent creates simplified representations of complex data patterns by storing key statistical parameters (mean, standard deviation, minimum, maximum) as copies of the underlying data distribution. These statistical copies enable rapid anomaly detection without requiring access to or processing of the original large volumes of historical data, reducing computational complexity while preserving adaptability
3Ease of operation
If all data records are stored and indexed for rapid retrieval, then forensic investigation capability is improved, but storage requirements increase
Solution Approach 1:
The patent extracts and stores only the essential statistical parameters (mean, standard deviation, minimum, maximum values) from the complete data records. This extraction enables rapid anomaly detection and forensic investigation by comparing new data against these extracted statistics, while significantly reducing storage requirements compared to retaining all original data records
Data Source
AI summary
A method and system for analysing data is disclosed. One or more data records are passed to a data analysis system. The data records comprised a plurality of data items and a first one of the data items is selected from the data items in the data record. A statistical model can be retrieved from a store in a computer system and the statistical model used to detect abnormal results from the selected data item and produce a data model. This statistical model is stored with the data record in the data base.