ML Quantification of Text Analytics Data Irregularity Impact

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

It is technically challenging to quantify and minimize the impact of irregularities in input text used to design and operate text analytics applications, which affects their performance.

Innovation Solution

A machine learning-based apparatus and method that generates irregularity feature vectors to identify and minimize lexical, morphological, parsing, semantic, and statistical irregularities in input text, allowing for the creation of normalized data and corresponding machine learning models that optimize application performance.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If irregularities in input text are not minimized, then development time and processing speed may be improved, but performance accuracy deteriorates

Engineering Contradiction:

Improvedevelopment timeVSAvoidperformance accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The system performs preliminary analysis of data irregularities and predicts their impact on performance before full processing begins. By generating irregularity feature vectors and predicting performance loss in advance, the system allows developers to make informed decisions about whether to invest time in data normalization, thus resolving the contradiction between development speed and accuracy

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system changes the parameter of data representation by generating irregularity feature vectors that quantify different types of irregularities (lexical, morphological, parsing, semantic, statistical). This transformation allows the system to assess and manage the trade-off between processing raw data quickly versus normalizing it for better accuracy

Inventive Principle:

Principle #35Parameter changes

2Measurement precision

If data normalization is performed to minimize irregularities, then performance accuracy is improved, but processing time and computational complexity increase

Engineering Contradiction:

Improveperformance accuracyVSAvoidprocessing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system applies partial normalization by selectively addressing only those irregularities that are predicted to have significant impact on performance. Rather than fully normalizing all data, the system uses regression models to identify and prioritize critical irregularities, thus achieving improved accuracy without the complete time cost of full normalization

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The system performs preliminary prediction of performance loss due to irregularities before committing to normalization processes. This advance assessment allows the system to justify and plan normalization efforts only when necessary, optimizing the balance between processing time and accuracy improvement

Inventive Principle:

Principle #10Preliminary action

3Reliability

If comprehensive irregularity analysis is performed, then data quality is improved, but device complexity and computational resources increase

Engineering Contradiction:

Improvedata qualityVSAvoidcomputational complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system segments the complex task of irregularity analysis into distinct components: lexical irregularities, morphological errors, parsing irregularities, semantic irregularities, and statistical irregularities. Each type is analyzed separately using specific feature extraction methods, making the overall complex process more manageable and computationally efficient while maintaining comprehensive data quality assessment

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system introduces irregularity feature vectors as an intermediary representation between raw text data and performance prediction. These vectors serve as a compact, structured intermediate form that captures essential irregularity characteristics without requiring full complex analysis of the original data, thus reducing computational complexity while preserving data quality information

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS11210471B2Machine learning based quantification of performance impact of data veracity

Publication Date: 2021.12.28 ACCENTURE GLOBAL SOLUTIONS LTD

AI summary

In some examples, machine learning based quantification of performance impact of data irregularities may include generating an irregularity feature vector for each text analytics application of a plurality of text analytics applications. Normalized data associated with a corresponding text analytics application may be generated for each text analytics application and based on minimization of irregularities present in un-normalized data associated with the corresponding text analytics application. An un-normalized data machine learning model may be generated for each text analytics application and based on the un-normalized data associated with the corresponding text analytics application. A normalized data machine learning model may be generated for each text analytics application and based on the normalized data associated with the corresponding text analytics application. A difference in performances may be determined with respect to the un-normalized data machine learning model and the normalized data machine learning model.