Shared Compression Dictionary for Low-Bandwidth Log Collection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing log collection systems face performance degradation due to high communication resource consumption caused by large text log data, limiting the number of log generation devices that can be managed by a single log collection device, especially in scenarios with narrow communication bandwidth or high transmission volumes.
Innovation Solution
A log collection system where the log collection server generates and shares a compressed dictionary with log generation devices, allowing them to perform text compression using this dictionary, thereby reducing the processing load on log generation devices and minimizing data size transmission.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If text log data is transmitted without compression, then the log collection device can receive complete log messages, but communication resource consumption increases and system performance degrades
Solution Approach 1:
The log collection device performs preliminary compression of log messages before transmission using LZW compression algorithm. This preliminary action reduces the data size that needs to be transmitted over the communication network, thereby reducing communication resource consumption while ensuring the complete log message can be reconstructed at the receiving end through decompression.
2Loss of energy
If lexicographic compression is performed at the log generation device, then communication resource consumption is reduced, but the processing load of the log generation device increases
Solution Approach 1:
Instead of performing compression at the log generation device (sender side), the invention inverts the approach by performing compression at the log collection device (receiver side) before transmission. This inversion shifts the processing load from resource-constrained log generation devices to the more capable log collection device, reducing communication resource consumption while minimizing the impact on log generation device performance.
3Adaptability or versatility
If one log collection device manages a large number of log generation devices, then system scalability is improved, but communication bandwidth requirements increase and performance degrades
Solution Approach 1:
The log collection device performs preliminary LZW compression on incoming log messages from multiple log generation devices before transmitting them. This preliminary compression action significantly reduces the total data volume that needs to be transmitted over the communication network, enabling the single log collection device to manage a larger number of log generation devices without degrading system performance or exceeding communication bandwidth limits.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
There is provided a log collection device including: a log receiver that receives a text log from a log generation device; a dictionary generator that generates a compressed dictionary for performing text compression based on the received text log; a dictionary transmitter that transmits the generated compressed dictionary to the log generation device and instructs performing of the compression process using the compressed dictionary on the text log transmitted after the transmission of the compressed dictionary; and a decompression processor that performs a decompression process using the compressed dictionary on the text log received after transmission of the compressed dictionary.