Edge Agent Data Compression for Reduced Transmission Volume
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing approach of sending large volumes of data from edge devices to a remote data center is time-consuming, expensive, and raises privacy concerns, leading to delays in detecting root causes of performance issues, which can impact real-time applications and damage an organization's reputation.
Innovation Solution
Implementing agents at edge devices that collect and process event information, summarizing it into relevant data, and sending only this summarized information to the data center, thereby reducing the amount of data stored and transmitted, and enabling faster issue detection and resolution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If large quantities of data are transferred to a remote data center, then data storage and processing capabilities are improved, but transmission time and cost increase
Solution Approach 1:
The system segments data processing into two parts: local processing at the edge device and centralized processing at the data center. The edge device performs initial data collection, filtering, and aggregation locally, then transmits only processed results to the data center. This segmentation reduces the volume of data transmitted while maintaining the benefits of centralized processing capabilities.
Solution Approach 2:
The edge device performs preliminary data processing actions before transmission to the data center. This includes local filtering of irrelevant data, aggregation of related data points, and pre-processing transformations. By completing these actions at the edge, the system reduces the quantity and improves the relevance of data transmitted to the remote data center.
2Loss of information
If all event information is sent to the data center, then comprehensive data availability is improved, but data privacy risks and storage costs increase
Solution Approach 1:
The system extracts and removes sensitive or irrelevant data at the edge device before transmission to the data center. By taking out unnecessary information and filtering data based on predefined criteria, the system reduces the exposure of sensitive information while maintaining the essential data needed for analysis and decision-making.
Solution Approach 2:
The edge device applies local data processing quality standards that prioritize data relevance and privacy protection. Each edge device can independently configure its filtering and processing criteria based on local requirements, ensuring that only data meeting specific quality and privacy standards is transmitted to the centralized data center.
3Loss of time
If data is processed locally at the edge device, then transmission costs and time are reduced, but processing complexity at the edge increases
Solution Approach 1:
The edge device performs self-service processing by autonomously filtering, aggregating, and preparing data for transmission without requiring constant intervention from the centralized data center. This self-service capability reduces transmission requirements while distributing processing intelligence to the edge, balancing local complexity with overall system efficiency.
Data Source
AI summary
Computer-implemented processes and systems described herein are directed to reducing volumes of data sent from edge devices to a data center. Each edge device runs an agent that collects event information generated by event sources of the edge device in a runtime interval. Each agent reduces the event information to relevant event information at the edge device in accordance with instructions received from a controller server of the data center. The relevant event information contains less information than the event information. Each agent forwards the relevant event information over the internet to external services executed at the data center, where the relevant event information is stored in a data storage device of the data center.