Edge-Based Load Shedding for Fast Data Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional load shedding methods in data stream management systems often reduce analysis accuracy by ignoring helpful input data, which is not suitable for real-time processing in fast data environments, especially when dealing with high-speed sensor data from IoT systems.
Innovation Solution
An edge-based adaptive load shedding method that offloads preprocessing operations from the analysis server to an edge machine, allowing it to process and filter raw data before transmitting only preprocessed data back to the server, thereby reducing the server's load and improving processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If conventional load shedding methods are used to reduce server load, then server resources are conserved, but analysis accuracy is reduced due to ignoring helpful input data
Solution Approach 1:
The system segments the data processing function by introducing an edge machine that performs preprocessing separately from the analysis server. The edge machine filters and preprocesses data locally, then transmits only necessary preprocessed data to the analysis server, dividing the processing workload and preserving both load management and analysis accuracy.
Solution Approach 2:
The edge machine acts as an intermediary between the data source and the analysis server. It receives raw data, performs preprocessing and filtering operations, and then transmits the processed data to the analysis server, mediating the data flow to reduce server load while maintaining data quality for accurate analysis.
2Productivity
If data is processed at the analysis server, then centralized analysis is simplified, but processing speed decreases due to data transmission distance and server bottleneck
Solution Approach 1:
The processing function is segmented between the edge machine (preprocessing) and the analysis server (analysis), allowing data to be processed closer to its source and reducing the transmission distance for critical data, thereby improving processing speed while distributing system complexity.
Solution Approach 2:
The edge machine performs preliminary preprocessing actions on data locally before transmitting it to the analysis server. This preliminary action reduces the amount of data that needs to be transmitted and processed centrally, improving overall processing speed by performing necessary operations up front.
3Quantity of substance
If all raw data is transmitted to the analysis server, then complete data availability is ensured, but network traffic and server load increase excessively
Solution Approach 1:
The edge machine extracts and removes unnecessary data through preprocessing and filtering operations before transmission to the analysis server. By taking out only the essential information and transmitting preprocessed data, the system reduces network traffic while ensuring data availability for analysis.
Solution Approach 2:
The edge machine extracts and removes unnecessary data through preprocessing and filtering operations before transmission to the analysis server. By taking out only the essential information and transmitting preprocessed data, the system reduces network traffic while ensuring data availability for analysis.
Data Source
AI summary
Provided is an edge-based load shedding system. The edge-based load shedding system includes an analysis server configured to determine to perform server or edge separation execution, transmit an operational module of an analysis application, issue a request to offload the transmitted operational module, and receive and analyze data obtained through preprocessing and an edge machine configured to receive and offload the operational module from the analysis server, preprocess raw data collected from a sensor by using the offloaded operational module, and transmit the data obtained through the preprocessing to the analysis server.