Network Data Characterization for Real-Time Unknown Malware Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing network security tools are inefficient against modern threats due to reliance on signatures and slow sandbox detection, requiring additional hardware and custom integration, and are unable to proactively detect unknown malware.
Innovation Solution
A system and method for network data characterization using machine-learning techniques to analyze network traffic in real-time, classifying content with confidence percentages, and integrating with existing security tools for proactive threat detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If signature-based detection and sandbox analysis are used, then detection accuracy for known malware is improved, but detection speed deteriorates and cannot keep up with modern threat volumes
Solution Approach 1:
The system segments the detection process into multiple specialized analyzers that work in parallel: signature-based analyzers for known threats, sandbox analyzers for behavioral analysis, and machine learning analyzers for unknown threats. Each analyzer processes a portion of the traffic simultaneously, maintaining both high accuracy and speed by dividing the detection workload across multiple specialized components rather than using a single slow comprehensive analysis method
Solution Approach 2:
The system applies partial action by using pre-filters to quickly eliminate obviously benign traffic before subjecting all remaining traffic to comprehensive analysis. This allows the system to maintain high detection accuracy for suspicious traffic while processing the entire traffic volume at high speed by not applying full analysis resources to every single file
2Reliability
If sandbox environments are used to examine malware behavior, then detection of new malware is improved, but processing time increases making it too slow for high-volume traffic
Solution Approach 1:
The system segments traffic processing by applying sandbox analysis only to specific subsets of traffic identified as suspicious by pre-filters, rather than analyzing every file in the traffic stream. This allows reliable behavioral analysis to be applied where needed while maintaining overall high processing speed by limiting sandbox usage to only the necessary portion of traffic
Solution Approach 2:
The system performs preliminary filtering and classification of traffic before sandbox analysis using lighter-weight signature-based and machine learning analyzers. This preliminary action identifies which files warrant sandbox examination, reducing the total number of files requiring time-consuming sandbox analysis while maintaining detection reliability for actual threats
3Adaptability or versatility
If multiple traditional security tools are deployed, then coverage of different threat types is improved, but system complexity and integration requirements increase
Solution Approach 1:
The system merges multiple security analysis functions into a single integrated platform that hosts multiple analyzers (signature-based, sandbox, machine learning) working together. This consolidation provides comprehensive threat coverage equivalent to multiple separate tools while reducing integration complexity by having all analyzers communicate through a unified architecture rather than requiring custom integrations between separate products
Solution Approach 2:
The system implements multi-functionality by designing a unified analyzer platform that can perform multiple detection functions (signature matching, behavioral analysis, machine learning classification) through a single system. This universal platform handles diverse threat types without requiring separate specialized tools, simplifying the overall security architecture while maintaining versatile threat coverage
4Speed
If pre-filters are used to reduce sandbox workload, then processing speed is improved, but detection capability is reduced to only previously observed patterns
Solution Approach 1:
The system segments the detection pipeline into multiple stages with different capabilities: pre-filters for speed, signature-based analyzers for known threats, and machine learning analyzers for unknown threats. Each segment handles a specific portion of the detection task, allowing the system to maintain high processing speed while preserving comprehensive detection capability through the combined effort of all segments
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
Described herein are embodiments of a system and method for network data characterization and/or classification that overcome the defects of the prior art. These and other advantages are achieved by a method for network data characterization. The method includes receiving network events, wherein at least some of the events include content, converting the received content into data messages, in which at least some of the data messages include the content and metadata describing an event and the content, routing the data messages to a plurality of analyzers based on specified message criteria to which each analyzer subscribes, each of one or more analyzers that received the routed data messages analyzing the content within the data messages in order to characterize the content, in which the one or more analyzers include at least one machine-learning analyzer that classifies the content with a confidence percentage that indicates the probability that the content is malign or the confidence that a prediction that the content is malign is correct, outputting the characterization results of the one or more analyzers, and comparing the output characterization results against a plurality of criteria to determine subsequent action to take based on the characterization results.