Dynamic Data Processing System for Event Span Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data collection methods, such as event-driven and periodic-pull models, face inefficiencies including resource wastage, scalability issues, and outdated data due to unnecessary API calls and static polling schedules, which decrease productivity and increase infrastructure costs.
Innovation Solution
Implementing a scalable and dynamic data processing system that uses machine learning to predict event spans and data changes, allowing for efficient packaging and handling of events or delayed data extraction based on probability thresholds, thereby optimizing resource usage and reducing unnecessary data retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If an event-driven model is used to collect data from sources, then real-time data updates can be detected, but numerous unnecessary events and notifications increase noise and consume excessive computing and networking resources
Solution Approach 1:
The patent creates a simplified copy or representation of events through event typing and categorization. Instead of processing every raw event notification, the system creates typed event objects that capture only the essential information needed for processing, reducing the computational overhead while maintaining real-time detection capability
Solution Approach 2:
The patent extracts only the relevant and actionable information from event notifications by implementing event filtering and typing mechanisms. The system takes out essential event attributes (type, source, timestamp, data) while discarding unnecessary notification overhead, thereby reducing noise and resource consumption while preserving real-time update detection
2Loss of energy
If periodic-pull model is used to collect data from sources, then resource usage can be reduced by avoiding unnecessary API calls, but data may become outdated due to static polling schedules
Solution Approach 1:
The patent implements dynamic polling intervals that adjust based on data source characteristics, event types, and system load conditions. Instead of using fixed static schedules, the system dynamically modifies polling frequency to match actual data change patterns, ensuring data freshness while optimizing resource usage by reducing unnecessary API calls during low-activity periods
Solution Approach 2:
The system incorporates feedback mechanisms where polling intervals are adjusted based on historical data change patterns, event frequency, and system performance metrics. This feedback loop allows the system to learn from past behavior and optimize polling schedules, maintaining data freshness while minimizing resource consumption through adaptive rather than static timing
3Reliability
If all events are processed individually in real-time, then complete event data can be captured, but system scalability is limited and the system can crash during event spikes
Solution Approach 1:
The patent merges multiple individual event processing operations into batch processing workflows. Events of the same type or from the same source are grouped together and processed collectively rather than individually, reducing the overhead of event handling operations while maintaining complete event data capture. This batching approach significantly improves system scalability during high-event-volume periods
Solution Approach 2:
The system implements periodic batch processing where events are collected over defined time windows or until a threshold count is reached, then processed together in periodic batches. This approach prevents system overload during event spikes by controlling the rate of processing operations while ensuring all events are eventually captured and handled, thereby improving both reliability and scalability
Data Source
AI summary
Examples described herein include systems and methods performing scalable and dynamic data processing and extraction. A first example method relates to processing events from a source. The method can include detecting an event generated by the source and predicting a probability of that event being part of a span including multiple events. The method can include waiting for the additional multiple events to occur within the predicted timeframe and, if occurring, packaging the events together for handling by a single dynamic function. Otherwise, the events can each be handled by separate dynamic functions. A second example method relates to performing dynamic data extraction from a source. The method can include waking up a function based on a regular poll interval, determining a probability of a data change at the source based on historical data extractions, and invoking an extraction function based on the probability of the data change.