Distributed Anomaly Detection Using Outlier Exposure Autoencoders
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current anomaly detection solutions for edge servers in 5G networks face challenges in leveraging knowledge from multiple edges, preserving user privacy, and managing variable data volumes, while existing solutions either compromise on accuracy or incur high communication overhead.
Innovation Solution
A distributed anomaly detection system using Federated Learning (FL) and Outlier Exposure (OE) capability, where an autoencoder is trained to minimize reconstruction error on unlabeled network data and maximize it on labeled attack data, allowing edge servers to collaborate and detect anomalies with reduced network overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If centralized anomaly detection is used, then detection accuracy is improved, but communication overhead and data privacy issues worsen
Solution Approach 1:
The system segments the anomaly detection process by deploying edge servers distributed across the network. Each edge server independently performs anomaly detection locally on network traffic, eliminating the need to centralize all data for processing. This segmentation reduces communication overhead while maintaining detection accuracy through distributed intelligence.
Solution Approach 2:
The patent introduces an intermediary mechanism where edge servers collaborate through a coordinated framework that enables knowledge sharing without direct data centralization. The system uses federated learning concepts where models are trained collectively across edges while preserving local data privacy, thus reducing communication requirements compared to full centralized processing.
2Measurement precision
If edge servers collaborate to detect anomalies, then detection accuracy is improved, but data privacy and communication costs worsen
Solution Approach 1:
Each edge server maintains local autonomy in processing network traffic data. The system allows edge servers to train and update anomaly detection models independently using local traffic patterns, preserving user privacy by keeping data localized. This local quality approach enables accurate detection while preventing data leakage to central authorities.
Solution Approach 2:
Instead of sharing raw data centrally, the system uses model copying and parameter sharing mechanisms. Edge servers receive updated model parameters from collaborators and apply them locally, effectively copying the intelligence without copying sensitive data. This enables collaborative improvement of detection accuracy while maintaining strict data privacy.
3Measurement precision
If more data is collected for training, then anomaly detection performance is improved, but communication overhead and processing burden worsen
Solution Approach 1:
The system implements dynamic data collection and model training where edge servers adaptively select which data to process and which models to update based on local conditions. This dynamic approach allows the system to improve detection performance by leveraging available data where most useful without attempting to collect and process all possible data, thus reducing overall complexity.
Solution Approach 2:
The patent applies partial action by having edge servers process only the portion of data most relevant to local anomaly patterns rather than processing all network traffic uniformly. This selective processing improves detection performance for local conditions while reducing the total processing burden and communication overhead compared to comprehensive data collection approaches.
Data Source
AI summary
A method, system and apparatus for outlier exposure based anomaly detection for detecting anomalies in network traffic are disclosed. According to one or more embodiments, a central node (17) is configured to train an Outlier Exposure (OE)-based autoencoder using unlabeled network data and labeled network attack data where the OE-based autoencoder is trained to reconstruct input data with an objective that is configured to minimize a reconstruction error on unlabeled network data and to maximize the reconstruction error on labeled network attack data, use the trained OE-based autoencoder to determine a reconstruction error on network traffic, and compare the determined reconstruction error to a threshold to determine if local network traffic is an anomaly.