NLP Data Extraction for Big Data Analysis

Solution Overview

Problem

Analyzing machine-generated big data is challenging due to its voluminous and semi-structured nature, requiring significant user effort to create specific programs and rules for interpretation.

Innovation Solution

A data-driven big data mining and reporting system that uses natural language processing and machine learning to automatically identify relevant data attributes, extract additional attributes, and create dashboards or alerts without user input, enabling instant analysis and reporting of data trends and anomalies.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If specific programs and rules are created to analyze machine-generated data, then analysis accuracy is improved, but user workload and system complexity increase significantly

Engineering Contradiction:

Improveanalysis accuracyVSAvoidsystem complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The system performs self-service by automatically generating analysis programs and rules without requiring user creation. The patent implements automated program generation that analyzes machine-generated data sets and creates appropriate analysis programs and rules autonomously, eliminating the need for users to manually create these programs while maintaining high analysis accuracy

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system performs preliminary action by pre-generating analysis programs and rules before actual data analysis is needed. The patent describes a process where the system analyzes data sets in advance, creates appropriate analysis programs and rules, and stores them for future use, thereby simplifying subsequent analysis operations without requiring users to create programs at the time of analysis

Inventive Principle:

Principle #10Preliminary action

2Measurement precision

If specific programs and rules are created to analyze machine-generated data, then analysis accuracy is improved, but user workload increases significantly

Engineering Contradiction:

Improveanalysis accuracyVSAvoiduser workload

Core Design Contradiction:

Measurement precisionVSEase of operation

Solution Approach 1:

The system performs self-service by automatically generating analysis programs and rules without requiring user creation. The patent implements automated program generation that analyzes machine-generated data sets and creates appropriate analysis programs and rules autonomously, eliminating the need for users to manually create these programs while maintaining high analysis accuracy

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system performs preliminary action by pre-generating analysis programs and rules before actual data analysis is needed. The patent describes a process where the system analyzes data sets in advance, creates appropriate analysis programs and rules, and stores them for future use, thereby simplifying subsequent analysis operations without requiring users to create programs at the time of analysis

Inventive Principle:

Principle #10Preliminary action

3Measurement precision

If manual program creation is required for data analysis, then analysis precision is maintained, but time consumption increases

Engineering Contradiction:

Improveanalysis precisionVSAvoidtime consumption

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system performs preliminary action by pre-generating analysis programs and rules before actual data analysis is needed. The patent describes a process where the system analyzes data sets in advance, creates appropriate analysis programs and rules, and stores them for future use, thereby simplifying subsequent analysis operations without requiring users to create programs at the time of analysis

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system uses copying by reusing previously generated analysis programs and rules across multiple data analysis tasks. The patent implements a mechanism where once analysis programs and rules are created for one data set, they can be copied and applied to similar data sets, eliminating the need to recreate programs each time and significantly reducing time consumption while maintaining consistent analysis precision

Inventive Principle:

Principle #26Copying

Data Source

PatentUS10204146B2Automatic natural language processing based data extraction

Publication Date: 2019.02.12 CA TECH INC

AI summary

A data-driven big data mining and reporting system automatically identifies which data attributes to report from a first data set using natural language processing. The identified data attributes to report from the first data set is used to automatically extract additional data attributes to report from additional data sets so that the identified data attributes to report from the first data set and the extracted data attributes to report from the additional data sets can be reported without input from the end user.