Dynamic Log Storage Scoring for Cloud Cost Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The high costs associated with cloud-based storage for log files, particularly due to unnecessary storage and data transfer, are exacerbated by excessive logging from devices and applications, which does not provide sufficient benefits for maintenance and troubleshooting.
Innovation Solution
A method for dynamic computer log storage that scores log lines based on parameter count, novelty, and throughput rate, storing important lines in long-term cloud storage and less important ones in short-term local storage, thereby reducing cloud storage costs while preserving critical information for maintenance and troubleshooting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If all log data is stored in cloud-based storage, then complete log information is preserved for troubleshooting, but storage costs and data transfer costs increase significantly
Solution Approach 1:
The patent segments log data into different categories based on importance, novelty, and throughput rate. Critical logs with high scores are stored in cloud-based storage, while less important logs are stored locally or discarded. This segmentation resolves the contradiction by preserving only essential log information in expensive cloud storage while handling less important logs through cheaper local storage options.
Solution Approach 2:
The patent applies local quality by differentiating storage locations based on log characteristics. Different storage strategies are applied to different log segments: critical logs go to cloud storage, while routine logs are handled locally. This resolves the contradiction by optimizing storage cost for each log type according to its actual value for troubleshooting.
2Loss of information
If excessive logging is performed to capture all diagnostic information, then complete troubleshooting data is available, but storage costs and data processing overhead increase
Solution Approach 1:
The patent extracts and identifies only the essential and novel information from log data using a scoring mechanism that evaluates parameter count, novelty, and throughput rate. By taking out only the critical diagnostic information rather than storing all logs, the system maintains troubleshooting effectiveness while significantly reducing data volume and associated costs.
Solution Approach 2:
The patent changes the parameter of log selection from binary (store all or store none) to a continuous scoring system that evaluates multiple parameters including novelty, parameter count, and throughput rate. This allows the system to selectively store logs based on their diagnostic value, reducing overall data volume while preserving essential information.
3Ease of operation
If cloud-based storage is used for all log data, then log retrieval is convenient for maintenance, but data transfer costs and storage expenses increase
Solution Approach 1:
The patent applies partial action by transferring and storing only the essential portion of log data in cloud-based storage, rather than all log data. The scoring mechanism identifies which logs warrant cloud storage based on their diagnostic value, reducing data transfer costs while maintaining ease of retrieval for critical logs through cloud access.
Data Source
AI summary
Disclosed embodiments provide techniques for dynamic computer log storage. Log data is obtained, and templates are created from the log data. A parameter count is identified, and a score for each log line is computed, based on the parameter count, and/or additional information, such as the novelty and/or the throughput rate of the log line. Each log line is scored, and based on the score, a storage strategy is applied to the log line. When scoring indicates a potentially important log line, log lines that temporally preceded and followed that log line are saved in short-term log storage, so they can be reviewed in the event that troubleshooting is needed. In this way, the cost of long-term storage for log files can be reduced considerably, while still preserving important information that is useful for maintenance, troubleshooting, and operation of deployed devices and/or software applications and systems.