CDN Log Message Handler Prioritization During Outages
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Content delivery networks (CDNs) face challenges in efficiently managing log data storage and forwarding during external connection losses or degradations, leading to potential data loss of high-priority and low-priority log data.
Innovation Solution
Implementing a log message handler that prioritizes and buffers log data in faster memory storage before resorting to slower disk storage, with pre-allocated memory and disk space allocations based on log data type and priority, and pruning lower priority data when storage is full to ensure high-priority data retention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If log data is stored in faster memory storage during connection outages, then data retention reliability is improved, but storage capacity is limited and lower priority data must be pruned
Solution Approach 1:
The storage system is segmented into multiple priority levels (high priority and low priority log data). The memory buffer is divided into separate queues for different priority log data, allowing the system to retain critical high-priority data while managing limited capacity through structured segmentation of storage resources.
Solution Approach 2:
Different storage qualities are assigned to different log data types. High-priority log data receives premium fast memory storage space, while lower-priority data is stored in slower disk storage or pruned first when capacity is limited. This local quality differentiation ensures critical data retains reliability even when overall storage capacity is constrained.
2Reliability
If lower priority log data is pruned when storage is full, then high priority data retention is improved, but information loss increases
Solution Approach 1:
The system performs preliminary prioritization and classification of log data before storage decisions are made. By pre-allocating storage space for high-priority data and establishing pruning rules in advance, the system ensures that when storage capacity is exceeded, only lower-priority data is removed while critical high-priority data is preserved, minimizing information loss of important logs.
Solution Approach 2:
The system changes the retention parameter based on data priority. High-priority log data has a retention parameter set to maximum (always retain), while low-priority data has a conditional retention parameter (retain only when storage capacity permits). This parameter differentiation resolves the contradiction by making data loss selective rather than uniform.
3Productivity
If pre-allocated memory and disk space are used for different log data types, then storage management efficiency is improved, but device complexity increases
Solution Approach 1:
The log message handler automatically performs prioritization, classification, and storage allocation without external intervention. The system self-manages the complex task of distributing log data to appropriate storage locations (memory vs. disk, high-priority vs. low-priority queues) based on pre-defined rules, thereby improving storage management efficiency while encapsulating the complexity within the handler itself rather than requiring external complex management infrastructure.
Data Source
AI summary
A computer-implemented method on a device. The device has hardware including storage. The method includes obtaining log event data from at least one component or service on the device that is to be delivered to a component or service on a distinct device. Each log event data item has a priority. If a connection to an external location is lost, at least some of the log event data items are selectively stored in the storage, wherein the storing is based on priority of the log event data items. Otherwise, if the connection is not lost, at least some of the log event data items are sent to the at least one external location.