TLS Handshake Classification for Inline Malicious Session Blocking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems struggle to efficiently filter malicious behavior in Transport Layer Security (TLS) traffic due to encrypted payloads, making it difficult to detect and block command-and-control attacks using domain generation algorithms (DGAs).
Innovation Solution
An inline malicious detection system that classifies TLS sessions using features from handshake messages, such as cipher suites, server name indications (SNIs), and digital certificates, trained with deep learning models to identify potential threats and block malicious traffic during the handshake stage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If TLS encryption is used to secure communications, then security and privacy are improved, but the ability to detect malicious behavior is worsened due to encrypted payloads
Solution Approach 1:
The system performs classification and detection of malicious behavior during the TLS handshake stage, before the encrypted payload communication begins. By analyzing handshake messages (ClientHello, ServerHello, Certificate, etc.) and extracting features from these unencrypted or partially encrypted messages, the system identifies potential threats in advance, allowing blocking decisions to be made before the main encrypted data flow establishes.
Solution Approach 2:
The system introduces an intermediate classification layer that operates on TLS handshake messages as a mediator between the raw encrypted traffic and the final security decision. This intermediary process extracts meaningful features from handshake protocols and uses machine learning models to classify sessions as malicious or benign, enabling detection without requiring decryption of the full payload.
2Measurement precision
If traditional deep learning models (LSTM, CNN) are used to detect DGAs, then detection accuracy is improved, but computational overhead and latency increase
Solution Approach 1:
The system extracts only the most relevant features from TLS handshake messages, such as cipher suite characteristics, SNI patterns, certificate properties, and handshake message structures. By focusing on these specific extracted features rather than processing entire message streams or using complex architectural models, the system achieves effective DGA detection with reduced computational overhead and lower latency.
Solution Approach 2:
The system applies a simplified classification approach that processes only the essential handshake stage messages rather than analyzing the complete TLS session or full payload content. This partial action on the most informative portion of the traffic (the handshake phase) provides sufficient detection accuracy while significantly reducing processing time and resource requirements compared to comprehensive analysis methods.
3Productivity
If TLS sessions are analyzed in real-time during handshake, then detection speed is improved, but system complexity increases
Solution Approach 1:
The system segments the TLS detection process into distinct phases, focusing specifically on the handshake stage (ClientHello, ServerHello, Certificate, ServerKeyExchange, etc.). By dividing the analysis into this specific segment rather than processing the entire TLS session, the system achieves real-time detection capability while managing complexity through phased processing and targeted feature extraction from handshake messages only.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An inline malicious traffic detector captures handshake messages in a session with a security protocol. The inline malicious traffic detector comprises a classifier that generates a verdict for the session indicating malicious or benign. The classifier is trained on labelled sessions using custom features generated from handshake messages. Based on determining that the session is malicious using features of the handshake messages, the inline malicious traffic detector blocks the session.