Feature Value Generation for Anomaly Detection in Text Data

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Anomaly detection in computer systems faces challenges when dealing with non-numerical text data, such as log messages, as these data types are not suitable for existing anomaly detection methods and require conversion into vectors for effective anomaly detection.

Innovation Solution

A feature value generation device that digitizes non-numerical text data into vectors, utilizing a generator, learning unit, and detector to learn and detect anomalies during predefined periods, employing techniques like autoencoders and ID assignment for normalization.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If non-numerical text data is used directly in anomaly detection, then the original data format is preserved, but the data cannot be processed by existing anomaly detection algorithms

Engineering Contradiction:

Improvecompatibility with anomaly detection algorithmsVSAvoiddata format conversion

Core Design Contradiction:

Adaptability or versatilityVSLoss of information

Solution Approach 1:

The patent introduces an intermediary conversion process that transforms non-numerical text data into numerical vector representations. This intermediary step enables compatibility with anomaly detection algorithms while preserving the essential information content of the original text data through structured vectorization methods.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent changes the parameter representation of text data from symbolic/non-numerical form to numerical vector form. By converting text data into numerical vectors with specific dimensions and properties, the system enables mathematical processing while maintaining the semantic information through appropriate parameter selection and normalization.

Inventive Principle:

Principle #35Parameter changes

2Adaptability or versatility

If text data is converted into vectors for anomaly detection, then compatibility with detection algorithms is achieved, but the complexity of the processing system increases

Engineering Contradiction:

Improvecompatibility with anomaly detection algorithmsVSAvoiddata processing system

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent performs preliminary vectorization and normalization of text data before the anomaly detection process. By pre-converting text data into standardized vector formats and normalizing these vectors in advance, the system reduces the computational complexity during the actual detection phase and simplifies the overall processing pipeline.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS11829868B2Feature value generation device, feature value generation method, and program

Publication Date: 2023.11.28 NIPPON TELEGRAPH & TELEPHONE CORP

AI summary

A feature value generation device includes a generator configured to digitize non-numerical text data items collected at a plurality of timings from a target of anomaly detection, to generate vectors whose elements are feature values corresponding to the digitized data items; a learning unit configured to learn the vectors during a learning period so as to output a learning result; and a detector configured to detect, during a test period, for each of the vectors generated by the generator, an anomaly based on said each of the vectors and the learning result.