Data Anomaly Detection Using Statistical Distribution Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In the context of frequent data interactions between platforms, existing methods struggle to effectively detect data anomalies due to privacy protection processing, which obscures data meaning, and the increasing complexity of big data with constantly changing business rules, making it difficult to determine data anomalies using only business rules.
Innovation Solution
A method and device that compare current data packets with historical distributions using statistical methods such as Gaussian Mixture Models and clustering algorithms to identify anomalies, even when data is privacy-protected, by obtaining and analyzing data packets within a predetermined time frame and determining anomalies based on comparison results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If privacy protection processing is applied to data before transmission, then security and confidentiality are improved, but data meaning is lost and anomaly detection becomes difficult
Solution Approach 1:
The system performs preliminary actions by extracting statistical features (mean, variance, distribution characteristics) from data packets before privacy protection processing obscures the data. These statistical features are stored as reference data for later anomaly detection, enabling detection capabilities to be established before the data loses its original meaning
Solution Approach 2:
Statistical features serve as an intermediary between the original data and the privacy-protected data. Instead of directly analyzing the obscured data, the system compares statistical features of current data packets against historical statistical features, enabling anomaly detection without requiring access to the original meaningful data
2Device complexity
If only business rules are used to discover data anomaly, then detection methodology is simple, but workload is huge and detection is incomplete
Solution Approach 1:
The system replaces manual business rule-based anomaly detection with automated statistical analysis. By using statistical features (mean, variance, distribution patterns) and automated comparison algorithms, the system eliminates the need for complex, manually-maintained business rules while significantly improving detection efficiency and completeness
Solution Approach 2:
The system changes the detection parameters from business rule-based logical conditions to statistical parameters (mean, variance, distribution characteristics). This parameter transformation enables automated, scalable anomaly detection that adapts to changing data patterns without requiring updates to complex business rule sets
3Measurement precision
If statistical methods are used to compare data packets with historical distribution, then anomaly detection accuracy is improved, but computational complexity increases
Solution Approach 1:
The system extracts only the essential statistical features (mean, variance, distribution characteristics) from data packets, discarding unnecessary detailed information. This extraction approach maintains detection accuracy by preserving key statistical properties while significantly reducing computational complexity compared to analyzing complete data sets
Solution Approach 2:
The system performs partial analysis by focusing only on critical statistical parameters rather than comprehensive data examination. By calculating and comparing only mean, variance, and distribution characteristics, the system achieves sufficient detection accuracy with reduced computational effort compared to full data analysis
Data Source
Figure 1~2
Figure 3~4
Figure 5~6
AI summary
A method and device for determining a data anomaly. The method comprises: acquiring multiple data packages having the same data structure in a predetermined period of time; acquiring a historical distribution of historical data having the same data structure; comparing the multiple data packages with the historical distribution to obtain a comparison result; and determining whether a data anomaly exists according to the comparison result. The device corresponds to the method. By means of the method and device, a data anomaly in a currently acquired data package can be effectively determined.