Application Log Compression and Priority-Based Remote Transmission
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing software applications face inefficiencies in data transmission overhead and reliability when sending application log data to remote repositories, as they often transmit all data types equally, leading to increased network and processing resource usage without ensuring the priority and reliability of critical log data transmission.
Innovation Solution
Implement a dynamic compression and transmission method that prioritizes and compresses application log data based on its type, transmitting critical data with low latency and high reliability while delaying less critical data, using techniques such as immediate dispatch for error logs and batch processing for usage logs, and retrying failed transmissions until successful.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all log data is transmitted immediately without prioritization, then transmission reliability is improved, but transmission overhead and network resource usage increase
Solution Approach 1:
The patent segments log data into different priority levels (critical, important, normal, low) based on log type and application context. Critical logs are transmitted immediately with high reliability guarantees, while lower priority logs are delayed or compressed, reducing overall network resource usage while maintaining essential transmission reliability.
Solution Approach 2:
Different transmission qualities are applied to different log data segments based on their priority classification. Critical error logs receive immediate transmission with retry mechanisms, while usage statistics and non-critical logs are transmitted with lower priority or compressed formats, optimizing network resource allocation.
2Loss of energy
If compression is applied to all log data, then transmission overhead is reduced, but processing time and computational resources increase
Solution Approach 1:
Compression is selectively applied only to lower priority log data segments (normal and low priority logs), while critical error logs are transmitted uncompressed or with minimal compression to avoid processing delays. This segmentation approach reduces overall transmission overhead without significantly increasing processing time for critical data.
Solution Approach 2:
Instead of applying full compression to all log data, the system applies partial compression only where beneficial (lower priority logs), accepting that some transmission overhead remains for non-critical data in exchange for minimizing processing time impact.
3Reliability
If critical log data is prioritized for immediate transmission, then reliability of critical data delivery is improved, but transmission overhead for all data increases
Solution Approach 1:
The system segments transmission resources by priority level, dedicating immediate transmission capacity to critical logs while scheduling lower priority logs for delayed transmission. This ensures critical data reliability without requiring all data to be transmitted with the same high overhead.
Solution Approach 2:
The transmission system maintains continuous operation by immediately transmitting critical logs while simultaneously queuing and progressively transmitting lower priority logs. This continuous useful action ensures critical data reliability while efficiently utilizing network resources over time for all data types.
4Device complexity
If fire-and-forget transmission is used, then device complexity is reduced, but transmission reliability deteriorates
Solution Approach 1:
The transmission mechanism is segmented into different reliability levels: critical logs use a complex transmission protocol with retry logic and acknowledgment verification, while lower priority logs use simpler fire-and-forget transmission. This segmentation provides necessary reliability for critical data without applying excessive complexity to all transmission operations.
Solution Approach 2:
Different transmission quality levels are applied locally to different log types. Critical error logs receive high-quality transmission with retry mechanisms and delivery confirmation, while usage statistics receive lower-quality transmission without retries, optimizing the balance between reliability and complexity.
Data Source
AI summary
Certain aspects of the present disclosure provide techniques for committing log data in an application to a log data repository. An example method generally includes receiving, from an application, data to be committed to a remote storage location. A type of the received data is determined. The type of the received data is generally associated with a prioritization level and a compression mechanism to be used in committing the data to the remote storage location. An application execution context associated with the received data is determined. At a dispatch time associated with the prioritization level of the received data and the application execution context associated with the received data, a compressed data payload is generated and transmitted to the remote storage location. Generally, to compress the data payload, at least the received data is generally compressed based on the determined compression mechanism.