Dense Tensor Embeddings for Log Data Without Meaning Loss
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current techniques for analyzing software logs from network and system testing are non-systematic, inefficient, and fail to utilize valuable historical data, leading to resource wastage and incorrect modifications due to the inability to convert text-type software logs into a numeric format effectively for machine learning or AI models.
Innovation Solution
A data processing system transforms text-type software logs into high dimensional dense tensors by encoding alphanumeric data using vocabulary tokens, calculating adaptive sequence lengths, and training a sequential deep neural network to maintain meaningful information, reducing overfitting and conserving resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If text-type log data is converted to numeric format using traditional techniques, then machine learning models can process the data, but meaningful information is lost during conversion
Solution Approach 1:
The patent transforms text data into high-dimensional dense tensor representations by projecting text embeddings into a continuous vector space. This dimensional transformation preserves semantic meaning while enabling machine learning processing, resolving the contradiction between format compatibility and information retention.
Solution Approach 2:
The system changes the parameter representation from discrete text tokens to continuous dense vector representations. By transforming the data into a different parameter space (dense tensors instead of categorical text), the system maintains information integrity while achieving machine learning compatibility.
2Productivity
If traditional log analysis techniques are used, then analysis can be performed, but the process is non-systematic and inefficient
Solution Approach 1:
The patent replaces traditional mechanical/text-based log analysis with an automated deep learning system. By substituting manual analysis processes with neural network-based automated processing, the system achieves both systematic approach and high productivity.
Solution Approach 2:
The system performs self-service through automated training of deep neural networks on historical log data. The model learns patterns autonomously from past data, eliminating the need for manual systematic analysis while improving efficiency and productivity.
3Reliability
If historical data is not utilized in log analysis, then analysis can be performed on current data, but valuable historical information is wasted
Solution Approach 1:
The system performs preliminary action by pre-training deep neural networks on historical log data before analyzing current logs. This preliminary training enables the model to accurately process new data while utilizing historical information, improving reliability and preventing resource wastage.
Solution Approach 2:
The system incorporates feedback by continuously training models on historical data to improve their accuracy in analyzing current logs. The historical data serves as feedback that refines the model's performance, enhancing reliability while avoiding resource wastage through efficient pattern recognition.
4Measurement precision
If deep neural networks are trained on encoded data, then accurate embedding tensors are obtained, but computational resources are consumed during training
Solution Approach 1:
The system changes the data representation parameters by encoding text into dense vector formats that are more efficient for neural network processing. This parameter transformation enables accurate embedding tensor generation while reducing the computational burden compared to processing raw text data.
Data Source
AI summary
In some implementations, a device may obtain a training corpus, from a set of pre-processed log data, associated with an alphanumeric format. The device may encode the training corpus to obtain encoded data using a set of tokens. The device may calculate a sequence length based on a statistical parameter associated with the training corpus. The device may generate a set of input sequences and a set of target sequences based on the encoded data, where each input sequence and each target sequence has a length equal to the sequence length. The device may generate a training data set based on combining the set of input sequences and the set of target sequences. The device may train a deep neural network (DNN) using the training data set and based on one or more hyperparameters to obtain a set of embedding tensors associated with an embedding layer of the DNN.