Resource-Adaptive New Event Detection System
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing online new event detection (ONED) systems face inefficiencies due to quadratic time complexity, high resource requirements, and inability to handle bursty document streams, leading to slow processing speeds and inadequate handling of important documents in time-critical domains like intelligence gathering and news analysis.
Innovation Solution
A resource-adaptive system that classifies documents as important or non-important, stores only important information in memory, discards non-important documents when memory is full, and prioritizes important documents for processing, using indexing and compression methods to increase processing rates and manage resource limitations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing ONED systems compare a new document to all old documents, then new event detection accuracy is maintained, but processing time complexity becomes quadratic and processing speed becomes extremely slow
Solution Approach 1:
The patent segments the document set into two parts: a smaller subset of important documents that are retained in memory, and the remaining documents that are discarded. This segmentation allows the system to maintain detection accuracy by focusing on critical documents while dramatically reducing processing time by comparing new documents only against the retained subset rather than all historical documents.
Solution Approach 2:
The patent implements a mechanism to discard old documents from memory after they have served their purpose in event detection, and recover only the essential information needed for future comparisons. This approach reduces the growing memory burden and quadratic processing complexity while maintaining the ability to detect new events effectively.
2Measurement precision
If the system stores all documents in memory for comparison, then detection accuracy is maintained, but memory resource requirements become excessive and the system cannot handle high document arrival rates
Solution Approach 1:
The patent extracts and retains only the most important documents from the complete document set, separating them from the rest. This extraction process identifies which documents contribute most to detection accuracy and keeps only those in memory, thereby reducing memory consumption while preserving detection effectiveness.
Solution Approach 2:
The patent changes the parameter of document retention by introducing a threshold-based mechanism that determines which documents to keep in memory. By dynamically adjusting which documents are retained based on their importance or relevance parameters, the system optimizes the balance between memory usage and detection accuracy.
3Productivity
If the system processes all documents at high speed, then real-time detection capability is achieved, but the consumer cannot keep pace and important documents may be lost
Solution Approach 1:
The patent implements a feedback mechanism where the consumer's processing capacity influences the producer's output rate. When the consumer falls behind, the system adjusts by retaining more documents in memory or reducing the influx of new documents, ensuring that important documents are not lost while maintaining overall system throughput.
Solution Approach 2:
The patent creates a buffer (memory storage) beforehand to cushion the mismatch between production and consumption rates. This buffer allows the system to absorb peaks in document arrival and ensures that the consumer can process documents at its own pace without losing important information, thereby maintaining reliability despite varying speeds.
Data Source
AI summary
There are provided a system and method for resource adaptive, real-time new event detection. The method includes capturing, from among documents in a document streaming environment that includes a plurality of remote streaming document sources, first-story documents. The method further includes storing any of the documents determined to be the first-story documents in a memory. The method also includes classifying the documents as important or non-important based on pre-specified criteria. The method additionally includes storing, in the memory, only information of the documents classified as important, when the memory overflows. The method further includes discarding the documents that are classified as non-important when a document consumption rate from the memory is less than a document insertion rate into the memory. The method additionally includes removing the information of the documents classified as important from the memory subsequent to one of consumption and processing by a consumer.