Event Tweet Collection via Keyword Clustering and Active Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for collecting event-related tweets from social media platforms, such as Twitter, face challenges in precision and recall due to their reliance on location-based or keyword-based approaches, which often result in missing relevant tweets and returning false positives, especially during natural disasters where timely and accurate information is crucial.
Innovation Solution
A framework that constructs an initial set of keywords from a reference source, filters and clusters tweets, and updates keywords using active learning and machine learning algorithms to identify and classify relevant tweets, ensuring high precision and recall by grouping tweets into event-related and non-event-related clusters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If keyword-based methods are used to collect event related tweets, then the volume of collected data increases, but precision decreases due to false positives from insufficiently comprehensive keywords
Solution Approach 1:
The patent segments the keyword-based collection process into multiple stages: initial keyword set construction from reference sources, first-stage filtering using these keywords, candidate keyword generation from filtered tweets, and second-stage refinement. This multi-stage segmentation allows the system to collect large volumes of tweets while progressively improving precision by eliminating false positives at each stage.
Solution Approach 2:
The patent implements feedback mechanisms where collected tweets are analyzed to generate candidate keywords, which are then used to refine the keyword set for subsequent collection cycles. The system continuously learns from the collected data, adjusting keywords based on what proves relevant versus noisy, thereby maintaining high precision while collecting comprehensive tweet volumes.
2Measurement precision
If manually generated keywords are used for tweet collection, then precision may improve, but time consumption increases significantly
Solution Approach 1:
The patent performs preliminary action by automatically constructing an initial set of keywords from reference sources before the actual tweet collection begins. This pre-processing step establishes a foundation of relevant keywords without manual intervention, saving time while maintaining reasonable precision. The system then iteratively refines these keywords based on collected data.
Solution Approach 2:
The system performs self-service by automatically generating and refining its own keyword set through analysis of collected tweets. Instead of requiring continuous manual keyword generation, the system learns from the data it collects, automatically identifying relevant terms and phrases that emerge during the collection process and incorporating them into future keyword sets.
3Measurement precision
If location-based methods are used to collect tweets, then geographic relevance improves, but tweets from untagged locations or users outside the area are missed
Solution Approach 1:
The patent employs a universal keyword-based approach that functions independently of location data. The same keyword filtering mechanism applies to all tweets regardless of whether they contain location information, making the system multi-functional in handling both location-tagged and untagged tweets equally effectively.
Solution Approach 2:
The patent uses keywords as an intermediary mechanism to bridge the gap between geographic relevance and completeness. Instead of relying directly on location data, the system uses event-related keywords to identify relevant tweets, which indirectly captures geographic relevance while simultaneously including tweets from users who haven't tagged their locations but are discussing the event.
Data Source
AI summary
Described herein is a framework for collecting event related tweets. In accordance with one aspect of the framework, an initial set of keywords is constructed from a reference source. Tweets are collected from a messaging stream using the initial set of keywords for a first time window. The collected tweets are filtered to generate a candidate keywords set. The selected tweets of the candidate keywords set are grouped into a plurality of clusters. The clusters are classified into event related and non-event related clusters. The initial set of keywords is updated to obtain a new set of keywords.