TLS Handshake Anomaly Detection for Malicious Software Identification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing network security measures struggle to detect malicious software effectively, as it often avoids detection by abandoning TLS/SSL connection handshakes when a proxy is detected, making it difficult to differentiate between legitimate and malicious traffic.
Innovation Solution
Monitoring and analyzing unusual TLS/SSL handshake behavior, such as abandoned handshakes, to identify potential malicious software by processing telemetry and historical data, and applying techniques to determine if the behavior is indicative of malicious activity, which may involve flagging or quarantining affected endpoints.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If TLS/SSL interception proxies are used to detect malicious traffic, then encrypted traffic can be monitored, but malicious software can detect the proxy and abandon handshakes to avoid detection
Solution Approach 1:
The patent converts the harmful evasion behavior (abandoned handshakes) into a beneficial detection signal. By monitoring and analyzing handshake abandonment patterns, the system identifies malicious software that attempts to evade detection through proxy abandonment, thereby turning the evasion tactic into a detection opportunity.
Solution Approach 2:
The system implements feedback by continuously monitoring handshake behavior and using the observed patterns to improve detection accuracy. The analysis of telemetry data and historical information creates a feedback loop that refines the ability to distinguish between legitimate proxy abandonment and malicious evasion attempts.
2Reliability
If malicious software communicates with C&C servers using encrypted traffic, then communication security is maintained, but detection of malicious activity becomes difficult
Solution Approach 1:
The patent extracts specific handshake information (telemetry data) from the encrypted TLS/SSL traffic without requiring decryption of the full communication. By focusing on the handshake phase metadata rather than the encrypted payload, the system gains detection capability while maintaining traffic encryption.
Solution Approach 2:
The system changes the detection parameter from analyzing encrypted content to analyzing handshake metadata parameters. By monitoring handshake abandonment patterns, connection timing, and telemetry data rather than attempting to decrypt traffic, the system achieves detection of malicious activity without losing the benefit of encrypted communication.
3Measurement precision
If handshake behavior is monitored to identify malicious software, then detection accuracy improves, but false positives may increase
Solution Approach 1:
The system performs preliminary analysis of handshake behavior patterns before making definitive malicious software identification. By establishing baselines of normal handshake behavior and comparing observed patterns against these baselines, the system reduces false positives while maintaining detection accuracy.
Solution Approach 2:
The patent introduces historical information and telemetry data as intermediary evidence between raw handshake observations and malicious software identification. This intermediary analysis layer provides context and corroboration, reducing the likelihood of false positives by requiring multiple indicators before confirming malicious activity.
Data Source
AI summary
In one embodiment, a method includes identifying unusual behavior with respect to a handshake between a first endpoint and a second endpoint that are included in a network, and determining whether the unusual behavior with respect to the handshake indicates presence of malicious software. The method also includes identifying at least one of the first endpoint and the second endpoint as potentially being infected by the malicious software if it is determined that the unusual behavior with respect to the handshake indicates the presence of malicious software.