Unsupervised Theme Detection from Unstructured Data

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current methods for analyzing customer feedback from social media are limited by the use of predefined themes, which can miss new problems and overlook valuable feedback, as they rely on rule-based patterns or machine learning techniques that are not intuitive for human interpretation and struggle with the complexity of human language.

Innovation Solution

A system and method for automatically detecting discussion topics from unstructured feedback text using unsupervised techniques, which processes unstructured documents to discover themes, assign labels, and organize them in a hierarchy, allowing for the identification of frequently occurring terms and text patterns to create a category model for theme organization.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If predefined themes are used to analyze customer feedback, then the analysis process is structured and manageable, but previously unseen problems are not captured and valuable feedback is overlooked

Engineering Contradiction:

Improveability to capture new themesVSAvoidtheme detection system

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system performs preliminary unsupervised theme detection on a sample of customer feedback to automatically discover new themes before the main analysis. This preliminary action creates a dynamic theme set that combines predefined themes with newly discovered themes, enabling the system to adapt to previously unseen problems while maintaining structured analysis capabilities

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The theme detection system performs self-service by automatically discovering and labeling new themes without requiring manual intervention or supervision. The unsupervised learning algorithms autonomously identify patterns in customer feedback and generate meaningful theme labels, eliminating the need for continuous manual theme updates while maintaining high adaptability

Inventive Principle:

Principle #25Self-service

2Extent of automation

If rule-based patterns or machine learning techniques are used for theme mapping, then automated theme assignment is achieved, but the results are not intuitive for human interpretation

Engineering Contradiction:

Improveautomated theme detectionVSAvoidhuman interpretability

Core Design Contradiction:

Extent of automationVSEase of operation

Solution Approach 1:

The system introduces an intermediary layer of natural language processing that translates complex machine learning theme clusters into human-interpretable labels. This intermediary process generates descriptive theme names and keywords that bridge the gap between automated detection algorithms and human understanding, making the results intuitive while maintaining high automation

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system changes the parameter of theme representation from abstract cluster identifiers to natural language descriptions with associated keywords and hierarchical structures. This parameter transformation maintains the automated detection capability while significantly improving human interpretability through meaningful labels that reflect actual customer feedback content

Inventive Principle:

Principle #35Parameter changes

3Productivity

If unsupervised techniques are used for theme detection, then automated topic identification is achieved, but the cluster groupings are generally unintuitive to human interpretation

Engineering Contradiction:

Improvetheme detection speedVSAvoidinterpretability of results

Core Design Contradiction:

ProductivityVSEase of operation

Solution Approach 1:

The system performs preliminary processing of unsupervised theme clusters to generate human-interpretable labels before final output. This preliminary action includes selecting representative keywords, generating descriptive theme names, and organizing clusters hierarchically, which maintains the speed advantage of unsupervised techniques while improving interpretability

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system replaces the mechanical manual labeling process with an automated natural language generation system that creates interpretable theme descriptions from unsupervised clusters. This substitution maintains high productivity by automating the entire process while improving ease of operation through human-friendly output formats with meaningful labels and keywords

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS10372741B2Apparatus for automatic theme detection from unstructured data

Publication Date: 2019.08.06 CLARABRIDGE

AI summary

This apparatus provides a system and method of determining significant repeating themes in a collection of documents. The apparatus operates unsupervised and leverages a natural language processing mechanism supported with lexicon, synonym and taxonomy dictionaries to determine themes and establish their relevance using a two-level hierarchical structure. The apparatus also assigns meaningful names to identified themes and determines a set of rules that describe the theme such that it can be applied to categorize other documents outside of the collection as well.