Anomaly Detection via Index Reversing and Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection systems face high computational costs and complexity due to the need to search large numbers of database records, especially when using deep learning models or computationally expensive functions, making them impractical for real-world operations with large volumes of database transactions.
Innovation Solution
The system employs a first machine learning model to determine a search parameter for a record identified by a transaction, limiting the number of related record values considered by a second machine learning system, and uses index-reversing operations to efficiently store and retrieve records, thereby reducing the computational resources required for anomaly detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the system searches a large number of database records to detect anomalies using deep learning models or computationally expensive functions, then the anomaly detection accuracy is improved, but the computational cost and complexity increase significantly
Solution Approach 1:
The patent segments the anomaly detection process into two distinct phases: an offline training phase where deep learning models are pre-trained on historical data, and an online detection phase where pre-compiled rules and models are applied to new transactions. This segmentation allows computationally intensive operations to be performed offline, while online operations use lightweight pre-processed artifacts, thus reducing real-time computational complexity while maintaining detection accuracy.
Solution Approach 2:
The system performs preliminary actions by pre-compiling anomaly detection rules, pre-training machine learning models, and pre-processing transaction data during offline operations. These preliminary actions create ready-to-use artifacts that can be quickly applied during online anomaly detection, eliminating the need to perform complex computations in real-time and significantly reducing online computational complexity.
2Reliability
If the system processes all database transactions through complex anomaly detection operations, then the detection completeness is improved, but the processing time increases
Solution Approach 1:
The patent applies local quality by using different detection strategies for different transaction types and contexts. The system analyzes transaction characteristics and applies appropriate anomaly detection methods selectively - using simple rule-based checks for straightforward transactions and more complex model-based analysis only when necessary. This localized approach maintains detection completeness while reducing average processing time across all transactions.
3Adaptability or versatility
If the system uses deep learning models and computationally expensive functions for anomaly detection, then the detection capability is improved, but the resource consumption increases
Solution Approach 1:
The system performs preliminary actions by pre-training deep learning models and pre-compiling detection rules during offline operations when computational resources are abundant. The trained models and compiled rules are then stored and reused during online operations, eliminating the need to re-execute computationally expensive training and compilation processes. This preliminary action drastically reduces online resource consumption while maintaining advanced detection capability.
Solution Approach 2:
The patent uses copying by creating simplified copies or representations of complex anomaly detection logic in the form of pre-compiled rules and serialized model artifacts. These copies can be quickly loaded and applied during online operations without requiring the full computational overhead of the original complex models, thus reducing resource consumption while preserving detection capability.
4Measurement precision
If the system performs comprehensive record searches for every transaction, then the anomaly detection thoroughness is improved, but the search operations become slower
Solution Approach 1:
The system performs preliminary indexing and data organization operations offline, creating optimized data structures such as inverted indexes and pre-computed feature vectors. During online transaction processing, the system queries these pre-prepared structures rather than performing comprehensive searches from scratch, maintaining detection thoroughness while achieving significant speed improvements.
Data Source
AI summary
In some embodiments, a transaction request associated with a first transaction identifier may be obtained. A first database may be accessed to store a transaction entry in the first database based on the transaction request. A reversed key may be generated by reversing the first transaction identifier. A second database may be accessed to determine whether the second database comprises any records indexed by the reversed key. An aggregation record comprising the first value may be generated in the second database. In some embodiments, a second transaction identifier of an update message for the first database may be reversed to reconstruct the reversed key, and the second database may be searched with the reversed key to retrieve the aggregation record. The aggregation record may be updated based on the second value and sent to a destination after updating the aggregation record with the second value.