Abnormal Data Detection via Two-Group Significance Testing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In the context of data processing, especially in financial platforms, identifying abnormal data is challenging due to privacy protection processing that obscures data meanings and the increasing volume of data, making it difficult to determine exceptions like null values or outliers using service rules alone, leading to a heavy workload and incomplete identification.
Innovation Solution
A method and apparatus that utilize two-group significance tests (such as t tests, z tests, and t′ tests) on validated data groups and historical comparison data groups, with preprocessing to ensure homogeneity of variance, to effectively identify abnormal data by determining differences in population means and variances, and further divide data groups for more precise analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If service rules are used to identify abnormal data, then data exceptions can be detected, but the workload becomes heavy and identification becomes incomplete due to increasing data volume
Solution Approach 1:
The patent transforms the detection approach from rule-based service logic to statistical parameter-based significance testing. By changing the detection parameters from business rules to statistical metrics (mean, variance, significance levels), the system can efficiently handle large volumes of encrypted data without increasing workload, while maintaining comprehensive detection capability through mathematical rather than procedural means.
2Reliability
If privacy protection processing is performed on data, then data security is improved, but the service meaning of data is lost making it difficult to determine exceptions
Solution Approach 1:
The patent introduces statistical significance testing as an intermediary method between encrypted data and exception detection. Instead of directly analyzing encrypted data content (which loses service meaning), the system uses statistical parameters (mean, variance, significance test results) as intermediaries to detect abnormalities. This mediator approach allows detection of exceptions in encrypted data without requiring decryption or loss of privacy protection.
3Measurement precision
If data groups are divided for testing, then detection precision is improved, but the complexity of the detection process increases
Solution Approach 1:
The patent divides large data groups into smaller subsets for individual significance testing. This segmentation allows more precise detection by comparing each subset against historical data, improving the ability to identify localized abnormalities. The process complexity is managed through automated statistical computing that handles multiple subsets systematically, transforming manual complexity into algorithmic efficiency.
Data Source
AI summary
This specification describes techniques for detecting abnormal data in a data set. One example method includes obtaining, by a data processing platform, a to-be-validated data group including to-be-validated data corresponding to a predetermined feature; obtaining, by the data processing platform, a comparison data group including historical data associated with the to-be-validated data group, wherein the historical and the to-be-validated data are from a same data source; performing, by the data processing platform, a two-group significance test on the to-be-validated data group and the comparison data group to generate a test result; and determining, by the data processing platform, whether there is abnormal data in the to-be-validated data group based on the test result.