Real-Time Data Stream Classification Using Partial Reassembly
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods are unable to effectively classify data streams in real-time over telecommunications networks, particularly HTTP contents, due to limitations in processing out-of-order data blocks and distinguishing between cleartext and non-cleartext data, which is crucial for detecting malicious activities.
Innovation Solution
A method combining partial re-assembly of data streams with statistical analysis and Deep Packet Inspection (DPI) to classify data streams into groups, using entropy calculations and DPI analyses to differentiate between cleartext and non-cleartext data, and further sub-classifying non-cleartext data into encrypted and obfuscated categories.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full re-assembly of data streams is performed to enable accurate classification, then classification accuracy is improved, but processing time and computational resources increase significantly
Solution Approach 1:
The patent applies partial re-assembly by processing only the necessary portion of data blocks required for classification rather than waiting for complete re-assembly of the entire data stream. The classifier performs statistical analysis on available data blocks to determine if sufficient information exists for classification, enabling earlier processing decisions without requiring full data stream re-assembly.
2Productivity
If statistical analysis is performed on all data blocks regardless of order, then real-time processing is enabled, but classification accuracy deteriorates due to out-of-order data
Solution Approach 1:
The patent implements a preliminary check before statistical analysis to determine whether the received data blocks contain sufficient information for accurate classification. The classifier assesses the completeness and adequacy of the data blocks, and only proceeds with statistical analysis when the preliminary evaluation confirms that accurate classification can be achieved, thereby maintaining both real-time processing and classification accuracy.
3Measurement precision
If complex encryption detection methods are used to distinguish encrypted from obfuscated data, then detection accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent segments the detection process into multiple stages: initial statistical analysis to identify suspicious data streams, followed by targeted entropy calculations, and finally specialized detection methods for distinguishing encrypted from obfuscated data. This segmented approach allows complex detection methods to be applied selectively only when needed, reducing overall computational complexity while maintaining high detection accuracy.
Data Source
Figure 1
Figure 2~3
Figure 4
AI summary
The invention relates to a method of performing a real-time classification of a data stream exchanged between a client and a server over a telecommunications network, the method comprising: - receiving a data segment of the data stream, the data segment indicating a data offset; - updating a bitmap table with the received data segment, wherein the bitmap table comprises a number of blocks of data, the data blocks being ordered by block sequence numbers in the bitmap table; - after updating the bitmap table, performing measurements (603) on a complete data block to obtain at least one measurement result without waiting for all the data blocks having block sequence numbers that are lower than the first block sequence number to be completed; - performing (607) a statistical analysis based on measurement results obtained for several complete blocks; - classifying (608) the data stream into one group among a plurality of groups, based on the performed statistical analysis at least.