Textual Anomaly Detection Using WordPiece Tokenization and VAE
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies lack effective methods for detecting anomalies in large volumes of textual data and determining the normality of event sequences within broader datasets.
Innovation Solution
An anomaly detection tool employing a unique algorithm that utilizes a WordPiece tokenizer, variational autoencoder (VAE), and stochastic subgradient methods to identify anomalies by converting textual data into encodings, clustering observations, and using bootstrapping for accuracy estimation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional anomaly detection methods are used on large volumes of textual data, then processing time and computational resources increase significantly, but detection accuracy remains insufficient
Solution Approach 1:
The patent segments textual data into standardized tokens using a tokenizer, dividing the continuous text stream into discrete units that can be processed efficiently. This segmentation transforms unstructured text into a format suitable for machine learning processing, enabling both accuracy and efficiency improvements by working with standardized token sequences rather than raw text
Solution Approach 2:
The patent introduces a variational autoencoder as an intermediary model that learns the normal distribution of event sequences. This intermediary model acts as a mediator between raw event data and anomaly detection, encoding events into latent representations and comparing them against learned normal patterns, thereby improving detection accuracy while maintaining processing efficiency through the learned compression and reconstruction process
2Reliability
If comprehensive analysis of all textual data is performed, then detection reliability improves, but system complexity and computational load increase
Solution Approach 1:
The patent performs preliminary actions by pre-tokenizing and encoding event sequences before analysis, and by pre-training the variational autoencoder on normal event patterns. This preliminary processing creates a standardized foundation that improves reliability of subsequent anomaly detection while reducing the complexity of real-time analysis, as the heavy lifting of learning normal patterns is completed in advance
Solution Approach 2:
The variational autoencoder serves itself by automatically learning the distribution of normal events and generating its own reconstruction criteria for anomaly detection. The model self-adjusts its parameters through training on normal data, eliminating the need for manual threshold setting or complex rule-based systems, thereby improving reliability while keeping the system architecture relatively simple
Data Source
AI summary
Tool detects anomalies in textual data and determines event sequences normality against broader data. Representative event samples are gathered for source, and a top-level wordpiece tokenizer is built. Tokenizer is serialized and stored. Additional samples are gathered, and encodings pulled from inputs via the tokenizer. For a given variable, the algorithm either polls encodings in groups of time steps or pads encodings up to these time steps. A square matrix of observations is created, whose basis is expanded with a random matrix and added dimensions. The basis is expanded via a random projection. The matrices are then passed to a variational autoencoder. To minimize information loss when sending encodings to compressed latent space, stochastic subgradient methods are used. Upon convergence, the trained model is saved. Observed errors are bootstrapped on the holdout set. If new events fall outside tolerances set via bootstrap series is declared anomalous.