Per-Session Packet Collection for Big Data Log Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current log generation methods in big data systems are inefficient as they do not effectively minimize system load while capturing and recording vast amounts of network packets, lacking a method to match response and request messages effectively.
Innovation Solution
A log generating apparatus and method that utilizes per-session packet collection, featuring a session management unit for creating packet storage units for each session, a packet distribution unit for classifying and storing packets, and a log generation unit that reassembles and maps request and response packets to generate logs, reducing the need for separate processing of communication protocols like SYN and ACK.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If packets are captured and recorded in a network section to generate log data, then security abnormality detection capability is improved, but system load increases and log generation efficiency deteriorates
Solution Approach 1:
The patent segments the log generation process by creating separate packet storage units for each session (source IP, destination IP, source port, destination port). This segmentation allows parallel processing of multiple sessions simultaneously, improving log generation efficiency while maintaining comprehensive security monitoring capability.
Solution Approach 2:
The patent performs preliminary actions by pre-organizing packets into session-based storage units during packet capture, and pre-matching request-response pairs using protocol state machines. This preliminary organization reduces the processing burden during log generation, thereby improving efficiency without compromising detection capability.
2Reliability
If vast amounts of packets are captured and recorded, then security monitoring coverage is improved, but system resource consumption increases
Solution Approach 1:
The patent extracts only the essential information needed for security monitoring by matching request and response packets to generate structured logs. Instead of storing and processing all raw packet data, it extracts relevant security-related information, reducing system resource consumption while maintaining comprehensive monitoring coverage.
Solution Approach 2:
The patent changes the parameter representation from raw packet data to structured log entries with specific fields (source IP, destination IP, ports, protocols, etc.). This parameter transformation reduces data volume and processing requirements while preserving security monitoring effectiveness.
3Loss of information
If request and response packets are matched to generate logs, then log quality is improved, but processing complexity increases
Solution Approach 1:
The patent implements self-service mechanisms through protocol state machines that automatically track and match request-response pairs based on protocol characteristics. The system autonomously identifies matching packets without complex external coordination, improving log quality while managing processing complexity through automated protocol-aware matching.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A log generating method and apparatus based on per-session packet collection in a big data system is provided, in which the apparatus includes a session management unit including a plurality of packet storage units generated for each session established between a client and a server, a packet distribution unit that receives packets exchanged between a client and a server and distributes the packets to the plurality of packet storage units for each session, and a log generation unit that reassembles request packets and response packets stored in the packet storage units for each session into a request message and a response message, and generating a log by mapping information extracted from the request message and information extracted from the response message.