Keyword-Based Topic Modeling for Low-Compute Trend Discovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing topic discovery systems face challenges such as limited training data, lack of labeling resources, domain-specific terminologies, and reliance on human intervention, leading to inefficiencies and the need for unsupervised models that require high computing power and threshold tuning.
Innovation Solution
A keyword-based topic model utilizing unsupervised methods, including dependency parse trees and transformer sentence embeddings, automatically extracts trending topics without human intervention, leveraging DBSCAN for clustering and fine-tuning embeddings to generate deterministic results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If unsupervised topic models are used for automatic topic discovery, then human intervention is eliminated and productivity is improved, but computing power requirements increase and device complexity worsens
Solution Approach 1:
The patent segments the topic discovery process into distinct modules: text preprocessing, entity extraction, entity embedding generation, DBSCAN clustering, and topic identification. This modular segmentation allows each component to be optimized independently, reducing overall computational complexity while maintaining automated productivity
Solution Approach 2:
The patent introduces entity embeddings as an intermediary representation between raw text and topic clusters. By transforming entities into embedding vectors before clustering, the system reduces the computational burden of directly clustering high-dimensional text data, thus lowering device complexity while preserving automated topic discovery capability
2Measurement precision
If traditional supervised topic models are used, then topic discovery accuracy is improved, but labeling resources and time consumption increase
Solution Approach 1:
The patent implements self-service through automated entity extraction and embedding generation that does not require manual labeling. The system automatically processes text data, extracts entities, generates embeddings, and identifies topics through unsupervised clustering, eliminating time-consuming human labeling while maintaining topic discovery accuracy through deterministic DBSCAN clustering and co-occurring keyword analysis
3Measurement precision
If domain-specific terminologies are incorporated to improve topic relevance, then measurement precision is improved, but device complexity and model training difficulty increase
Solution Approach 1:
The patent changes the parameter representation by transforming domain-specific entities into embedding vectors with consistent dimensionalities. This parameter transformation allows domain terminologies to be incorporated without increasing model complexity, as the embeddings normalize different domain terms into a unified vector space that DBSCAN can cluster effectively
Data Source
AI summary
The present disclosure relates to a system, a method, and a product for topic discovery. The system includes a memory storing instructions; and a processor in communication with the memory. When the processor executes the instructions, the instructions are configured to cause the processor to: obtain text data, conduct pre-processing on the text data to obtain pre-processed text data, extract an entity list and a keyword list based on the pre-processed text data, generate an entity embedding list based on the entity list, clusterize the entity list based on the entity embedding list to obtain a plurality of entity clusters, each entity cluster comprising at least one entity, retrieve a co-occurring keyword list based on the plurality of entity clusters, the entity list, and the keyword list, and obtain a topic for each entity cluster of the plurality of entity clusters based on the co-occurring keyword list.