TLS Handshake Feature Analysis for Encrypted DGA Session Detection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing methods struggle to effectively filter malicious behavior in Transport Layer Security (TLS) traffic due to encryption of payloads, making it difficult to detect domain generation algorithms (DGAs) used in command-and-control attacks.

Innovation Solution

An inline TLS traffic detector captures and analyzes TLS handshake messages to generate features, using a TLS session classifier to determine malicious sessions, and blocks communications if detected, employing deep learning models like LSTM and CNN architectures to identify DGA-generated domain names.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If TLS encryption is used to secure communications, then security and privacy are improved, but the ability to detect malicious behavior in traffic is worsened

Engineering Contradiction:

ImprovesecurityVSAvoiddetection capability

Core Design Contradiction:

ReliabilityVSDifficulty of detecting and measuring

Solution Approach 1:

The patent extracts and analyzes TLS handshake metadata (SNI, cipher suites, protocol versions) from encrypted traffic without decrypting the payload. This allows detection of malicious patterns in the unencrypted handshake phase while preserving end-to-end encryption for the actual communication content.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The system performs detection during the TLS handshake phase before the encrypted session fully establishes. By analyzing handshake messages and metadata in advance, the system can identify and block malicious connections before they engage in encrypted malicious communication.

Inventive Principle:

Principle #10Preliminary action

2Measurement precision

If deep learning models like LSTM and CNN are used to detect DGAs, then detection accuracy is improved, but processing time and computational complexity are worsened

Engineering Contradiction:

Improvedetection accuracyVSAvoidprocessing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system applies deep learning models selectively to handshake metadata and SNI extraction rather than analyzing complete encrypted traffic streams. This partial application of complex models reduces processing time while maintaining detection accuracy for DGA patterns in domain names and handshake characteristics.

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The detection system segments traffic analysis into distinct phases: extracting unencrypted handshake metadata, analyzing SNI fields, and applying deep learning models to specific features. This segmentation allows parallel processing of different traffic components, reducing overall processing time while maintaining high detection accuracy.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentEP4145768B1Inline detection of encrypted malicious network sessions

Publication Date: 2025.11.26 PALO ALTO NETWORKS INC

Figure 1
Figure 2
Figure 3

AI summary

An inline malicious traffic detector captures handshake messages in a session with a security protocol. The inline malicious traffic detector comprises a classifier that generates a verdict for the session indicating malicious or benign. The classifier is trained on labelled sessions using custom features generated from handshake messages. Based on determining that the session is malicious using features of the handshake messages, the inline malicious traffic detector blocks the session.