Log File Compression via Neural Embeddings for Security
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data compression techniques for log files fail to recognize and compress sensitive or uncommon information, leading to potential unauthorized disclosures and increased memory and storage overhead, as they treat all information equally, including common and sensitive data.
Innovation Solution
A neural network model is trained to generate embeddings for log files, specifically identifying and compressing uncommon and sensitive information, creating distinct embedding vectors that obfuscate records, reducing file size and enhancing security by preventing generic decoding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If current data compression techniques are used for log files, then file size is reduced, but sensitive information is not properly protected and unauthorized disclosures may occur
Solution Approach 1:
The patent applies local quality by treating different portions of log file data differently during compression. Common information is compressed using standard techniques, while sensitive or uncommon information is identified and processed separately with enhanced protection measures, including potential encryption or specialized encoding, ensuring that each data portion receives the appropriate level of security treatment based on its sensitivity classification
Solution Approach 2:
The patent segments log file data into distinct categories (common information versus sensitive/uncommon information) and applies different compression and protection strategies to each segment. This segmentation allows the system to maintain high compression ratios for routine data while simultaneously providing robust security protection for sensitive portions, resolving the contradiction between compression efficiency and security reliability
2Device complexity
If all information in log files is treated equally during compression, then processing is simple, but sensitive information cannot be distinguished and protected
Solution Approach 1:
The patent implements preliminary action by pre-training the neural network model on extensive log file data before actual compression operations. During this training phase, the model learns to automatically distinguish between common and sensitive information patterns. This preliminary training enables the system to efficiently identify and protect sensitive data during subsequent compression operations without adding significant real-time processing complexity
Solution Approach 2:
The patent introduces a neural network model as an intermediary between the raw log data and the compression process. This intermediary automatically analyzes and classifies information sensitivity, enabling the system to differentiate between common and sensitive data without requiring complex manual processing rules, thus maintaining processing simplicity while achieving reliable sensitive information protection
3Productivity
If neural network model is trained to identify uncommon information, then compression efficiency improves, but training time and computational resources increase
Solution Approach 1:
The patent applies partial action by training the neural network model to focus specifically on identifying uncommon and sensitive information patterns rather than attempting to process every possible data variation. The model is trained on representative samples of log file data that contain various types of sensitive information, enabling it to generalize effectively. This approach achieves high compression efficiency for sensitive data without requiring exhaustive training on all possible data scenarios, thereby reducing training time and computational resource requirements
Data Source
AI summary
In some implementations, a device may train the model to generate embeddings for log files associated with an application, and to enable the model to generate embeddings for sensitive information included in a set of training log files. The device may receive a log file associated with the application. The device may generate a compressed log file including a set of embedding vectors associated with records included in the log file, where a record that includes sensitive information is associated with one or more embedding vectors for the sensitive information and one or more embedding vectors for other information included in the record. The device may store the compressed log file including the set of embedding vectors where a size of the compressed file is less than a size of the log file, and the embedding vectors obfuscate the records included in the log file.