Unsupervised Trending Term Detection in Data Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for tracking emerging topics in data streams, such as trends in politics, sports, and world events, are ineffective in identifying short-lived or newly emerging topics and fail to track the evolution of topics over time, relying on keyword-based searches that require prior knowledge of keywords and are not suitable for newly created words or dynamic vocabulary.
Innovation Solution
An unsupervised algorithm that analyzes data streams to identify trending terms and relationships without reference to a library of pre-defined terms, using frequency and co-occurrence analysis to detect statistically significant changes and communicate identified trends to users through a processor-based system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If keyword-based searches are used to track emerging topics, then the search can identify known topics using predefined keywords, but the method cannot identify newly created words or terms and requires prior knowledge of keywords
Solution Approach 1:
The system performs self-service by automatically learning and updating its vocabulary from data streams without requiring external supervision or predefined keyword lists. The unsupervised algorithm continuously adapts to new terms and topics as they emerge in the data, enabling the system to identify newly created words and terms autonomously
Solution Approach 2:
The system implements dynamics by making the keyword vocabulary dynamic rather than static. Instead of relying on fixed predefined keywords, the system continuously updates its understanding of relevant terms based on emerging patterns in the data streams, allowing it to adapt to newly created words and evolving topic terminology
2Reliability
If supervised algorithms are used to compile topic lists, then the system can make inferences based on previously-identified keywords, but the system cannot effectively track short-lived or newly emerging topics
Solution Approach 1:
The system performs preliminary action by proactively monitoring data streams for emerging patterns before topics become well-established. The unsupervised algorithm detects nascent trends and newly emerging topics in real-time, enabling early detection and tracking of short-lived topics before they fade from public discourse
Solution Approach 2:
The system performs self-service by automatically adapting its topic detection capabilities without requiring supervision or retraining. The algorithm continuously learns from incoming data streams, enabling it to reliably identify and track emerging topics as they develop without external intervention or predefined topic frameworks
3Productivity
If keyword-based searches are used, then the system can track topic occurrence frequency, but the system cannot track the evolution of topics over time
Solution Approach 1:
The system implements dynamics by transitioning from static keyword matching to dynamic term evolution tracking. The unsupervised algorithm monitors how terms and topics evolve over time, capturing the transformation and development of topics as they progress through different stages in the data streams
Solution Approach 2:
The system adds another dimension by incorporating temporal evolution analysis alongside frequency tracking. Instead of merely counting keyword occurrences, the system analyzes how topics transform and develop over time, providing a multi-dimensional view that includes both quantitative frequency data and qualitative evolutionary patterns
Data Source
AI summary
Methods, apparatus, and systems for analyzing data trends are described herein. The present disclosure includes the identification of trending terms in data through the use of an unsupervised algorithm. Trending terms are identified and counted during a first and second time period without reference to a library of pre-defined terms, along with at least one reason for using such trending terms. The set of trending terms and the at least one reason for use of the trending terms are displayed to a user.