Edge Node Agent Filtering and Enrichment for Sensor Data Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The vast volume of data generated by edge devices is resource-intensive to collect and process, with much of it being irrelevant or needing enrichment before being used by downstream systems, leading to inefficient data transmission and processing.
Innovation Solution
Implementing edge agents at edge nodes to perform data filtering, aggregation, and enrichment before sending data to central computing points, reducing data volume and enhancing its usability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If edge devices send all raw data to downstream systems, then complete data is available for processing, but data transmission volume and processing resources are excessively consumed
Solution Approach 1:
The edge agent performs preliminary data processing actions at the edge device before data leaves the edge. This includes filtering out irrelevant data, aggregating related data points, and enriching data with contextual information locally. By preparing data in advance at the source, the system reduces the volume of data needing transmission and processing at downstream systems, thereby conserving computational resources while maintaining data completeness for relevant information
Solution Approach 2:
The edge agent extracts and removes irrelevant or redundant data from the complete data set generated by edge devices. By identifying and eliminating unnecessary data points during the filtering process, the system transmits only the essential and relevant data to downstream systems, reducing transmission volume and processing requirements while preserving the integrity of meaningful information
2Loss of information
If edge devices send all raw data to downstream systems, then no data is lost, but data transmission volume increases
Solution Approach 1:
The edge agent extracts and removes irrelevant or redundant data from the complete data set generated by edge devices. By identifying and eliminating unnecessary data points during the filtering process, the system transmits only the essential and relevant data to downstream systems, reducing transmission volume and processing requirements while preserving the integrity of meaningful information
Solution Approach 2:
The edge agent combines multiple related data points into aggregated data structures. By merging redundant or correlated data elements into consolidated representations, the system reduces the overall volume of transmitted data while maintaining the essential information content needed for downstream processing and analysis
3Reliability
If edge devices send unprocessed data to downstream systems, then all original data is preserved, but data usability is reduced
Solution Approach 1:
The edge agent performs preliminary data enrichment actions at the edge device before data leaves the edge. This includes adding contextual information, metadata, and relevant attributes to raw data points locally. By enhancing data with additional useful information in advance, the system improves data usability for downstream systems without compromising the accuracy of original measurements, making the data more immediately actionable and easier to process
Solution Approach 2:
The edge agent acts as an intermediary between the edge device and downstream systems. It transforms raw data into an enriched format by adding contextual information and metadata while preserving the original data integrity. This intermediary processing layer enhances data usability for downstream applications without losing the accuracy of source measurements, bridging the gap between raw sensor data and actionable insights
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Edge devices can provide large amounts of data, only some of which may be relevant to a downstream system. In addition, edge device data may need to be processed, including being enriched with additional data, before being used by the downstream system. Disclosed techniques provide an edge agent that receives edge device data, including sensor data. The edge agent performs processing on the edge device data, which can include operations such as filtering and aggregation, as well as enriching the edge device data with values for attributes in addition to attributes and associated values that are present in edge device data sent from the edge device to the edge agent. The use of the edge agent and its associated actions can reduce a data volume sent to the downstream system, as well as making the edge device data more immediately useable by the downstream system.