Log Data Encoding System for Bandwidth Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The transfer of large application log data across networks faces issues with high network bandwidth usage and data lag, exacerbated by exponential growth when debug or trace is enabled, necessitating a solution to reduce bandwidth utilization and enhance transfer efficiency.
Innovation Solution
A system that encodes log data by matching strings against stored signatures, transferring only the deviation and signature identifier, allowing for reduced payload size and efficient compression, thereby minimizing data transferred while ensuring full data reconstruction at the destination.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large application log data is transferred over the network to the server, then the data can be analyzed by data analytics tools, but the network bandwidth usage increases and data lag occurs
Solution Approach 1:
The patent extracts only the essential information from log files by identifying and transferring only unique log entries and their deviations from templates, rather than transferring the complete log data. This extraction process reduces the amount of data transmitted over the network while preserving the ability to analyze application behavior, directly resolving the contradiction between maintaining data analysis capability and reducing network bandwidth usage.
Solution Approach 2:
The patent creates a simplified representation of log data by generating templates from unique log entries and then copying only the necessary deviation information rather than the entire log data. This copying approach allows the destination system to reconstruct the essential log information with minimal data transfer, addressing the bandwidth consumption issue while preserving analytical value.
2Reliability
If large application log data is transferred over the network, then the data can be analyzed, but data lag increases
Solution Approach 1:
By extracting only the essential log information and transferring minimal data representations (templates and deviations), the system significantly reduces transfer time and eliminates data lag while maintaining the ability to perform comprehensive data analysis at the destination.
Solution Approach 2:
The system performs preliminary processing locally by identifying unique log entries and generating templates before transfer. This preliminary action reduces the amount of data that needs to be transmitted, thereby reducing transfer time and eliminating data lag while preserving analysis capability.
3Loss of information
If debug or trace is enabled on an application, then the application can provide detailed logging information, but the application log size grows exponentially
Solution Approach 1:
The patent extracts only the unique log entries and their deviations from templates, filtering out redundant information. This extraction process maintains the detailed logging information provided by debug/trace modes while dramatically reducing the quantity of data that needs to be transferred and stored, resolving the contradiction between logging detail and data volume.
Solution Approach 2:
The system changes the representation parameters of log data from complete raw entries to simplified template-based representations. This parameter transformation maintains the informational content of detailed logging while reducing the data quantity to manageable levels, addressing the exponential growth problem.
Data Source
AI summary
According to one general aspect, a non-transitory computer readable medium includes instructions that, when executed by at least one processor, cause a computing device to read a string of a log file for an application, where the log file comprises multiple strings of log data, compare the string to signatures stored in a memory to find a matching signature, where each of the signatures is encoded with a signature identifier (ID), determine a deviation between the string and the matching signature, encode the string with the signature identifier (ID) of the matching signature and the deviation, and transfer the string to a destination computing device using the signature identifier (ID) of the matching signature, the deviation, and a timestamp of the string.