Aggregated Dataset Trigger Scheduling for Network Congestion Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data ingestion from multiple external sources increases network traffic, leading to overloaded networks and reduced performance.
Innovation Solution
A computer system that ingests data from multiple sources based on predefined trigger conditions, halts data ingestion when thresholds are met, and aggregates data before generating a unified dataset.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is continuously ingested from multiple external sources, then data completeness is improved, but network traffic increases causing overload
Solution Approach 1:
The system performs preliminary actions by ingesting data from external sources in advance based on trigger conditions (such as data availability or time-based triggers) before network congestion occurs. This allows data to be prepared and staged for aggregation without causing continuous network traffic, resolving the contradiction between data completeness and network overload.
Solution Approach 2:
The system implements periodic data ingestion cycles with halt periods between them. During each cycle, data is ingested from multiple sources, then the system halts ingestion to aggregate the collected data. This periodic on-off pattern ensures data completeness is achieved over time while preventing continuous network traffic that would cause overload.
2Object-affected harmful factors
If data ingestion is halted to reduce network traffic, then network performance is improved, but data processing delay increases
Solution Approach 1:
The system performs preliminary data collection during periods when network conditions are favorable, storing data locally before aggregation is needed. This preliminary action reduces the need for urgent data retrieval later, minimizing processing delays while allowing network traffic to be reduced when appropriate.
Solution Approach 2:
The system maintains continuity of useful action by performing data aggregation locally using previously ingested data during halt periods. This ensures that data processing continues without interruption even when network traffic is halted, preventing delays while improving network performance.
3Loss of time
If data from multiple sources is aggregated frequently, then data freshness is improved, but computing resources are consumed
Solution Approach 1:
The system performs data aggregation periodically at predetermined intervals or when trigger conditions are met, rather than continuously. This periodic aggregation maintains data freshness by updating the aggregated dataset at regular intervals while avoiding the continuous computing resource consumption that would result from frequent real-time aggregation.
Solution Approach 2:
The system performs preliminary data ingestion and local processing before aggregation is required. By having data ready in advance through preliminary actions, the actual aggregation process can be performed more efficiently with reduced computing overhead, balancing data freshness with resource consumption.
Data Source
AI summary
A computer system comprises and a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to ingest data from at least one first external data source in response to a first trigger condition; halt the ingesting of data from the at least one first external data source; ingest data from at least one second external data source in response to a second trigger condition; halt the ingesting of data from the at least one second external data source; and prior to a third trigger condition, aggregate the data ingested from the at least one first external data source and the data ingested from the at least one second external data source to generate an aggregated dataset. The first external data source may include a machine learning module trained to predict when network traffic will likely drop below a first threshold.