Adaptive Sensor Data Compression for Bandwidth-Limited Cloud Acquisition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional SCADA systems face challenges in efficiently storing and transmitting large amounts of time-series data from industrial automation systems, particularly due to bandwidth limitations when using cloud platforms, and existing compression methods either result in large data sizes or loss of original data quality.
Innovation Solution
Implementing a neural autoencoder-based data compressor and decompressor in field devices and cloud infrastructure, which learns and adapts compression algorithms using Big Data technology, allowing for efficient bandwidth use by transmitting only compressed data during normal operation and raw data upon deviations, and automatically updating representations to accommodate changing data characteristics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If lossless compression is used to store and transmit sensor data, then data quality and precision are maintained, but bandwidth consumption increases and data size becomes large
Solution Approach 1:
The system applies partial lossless compression by selectively transmitting only the residual error data (difference between original and compressed data) when deviation is detected, rather than transmitting all original data. This partial transmission of critical data maintains precision while reducing bandwidth consumption.
Solution Approach 2:
The system dynamically changes compression parameters by switching between lossy compression mode (for normal operation) and lossless compression mode (when deviation is detected). The comparison threshold and compression intensity are adjusted based on data characteristics and deviation detection results.
2Loss of energy
If lossy compression is used to reduce data size and bandwidth usage, then bandwidth consumption decreases, but data quality and ability to reconstruct original data deteriorates
Solution Approach 1:
The system implements feedback by comparing decompressed data with original sensor data using a comparison unit. When the difference exceeds a threshold, the system detects deviation and automatically switches to lossless compression mode, ensuring data quality is maintained when necessary.
Solution Approach 2:
The system prepares for potential data quality loss by having lossless compression capability ready as a backup. When deviation is detected in real-time, the system can immediately switch to lossless mode to cushion against further data quality degradation.
3Adaptability or versatility
If compression algorithms are manually configured by line builders or end users, then customization to specific data characteristics is possible, but configuration effort and required domain knowledge increase
Solution Approach 1:
The system performs self-service by automatically learning optimal compression algorithms and parameters from the data itself. The machine learning model adapts to data characteristics without manual configuration, and the system automatically adjusts compression settings based on real-time data analysis.
Solution Approach 2:
The system performs preliminary learning and adaptation of compression parameters before actual data compression begins. The machine learning model is trained on historical data in advance, preparing optimized compression settings that are then applied automatically during operation.
4Measurement precision
If all sensor data is transmitted as raw uncompressed data to cloud infrastructure, then data quality is maintained, but bandwidth consumption increases significantly
Solution Approach 1:
Instead of transmitting all original sensor data, the system transmits only the residual error data (difference between original and compressed data) when deviation is detected. This partial transmission maintains data quality for critical events while dramatically reducing overall bandwidth consumption.
Solution Approach 2:
The system extracts only the essential information (residual errors or deviation indicators) from the original data stream for transmission to the cloud, leaving out redundant data that can be reconstructed locally using the learned compression model.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Method, field device and cloud infrastructure for data acquisition Sensor data (SD) are compressed on field devices (SD) using a representation (R). The field device immediately decompresses the compressed data (CD) in order to detect a deviation. If there is a deviation, then a cloud storage (ST) receives the sensor data as raw uncompressed data (RUD). A cloud component (b2) receives a trigger signal (T) from the field device, indicating that the representation used by the field device for compression does not sufficiently describe the sensor data. The cloud component then learns a new representation (NR) by retrieving and analyzing all data (AD) stored in the cloud storage. The method and field device provide robust, compression-based data acquisition. They improve quality and precision of the data captured by the field devices. As the representation in the field device can be updated, it becomes possible to accommodate changes in the device setup. The cloud infrastructure provides automatic learning of the representation in the cloud. This benefits from utilizing all historic data of all field devices and saves resources on the field devices. For learning the new representation, cloud-based Big Data technology can be utilized towards building a new representation that can be automatically redeployed to the field devices.