Sensor Data Compression for Bandwidth-Limited IP Transport
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current technologies face challenges in efficiently transmitting and processing the large volumes of mixed qualitative and quantitative sensing data from edge devices, such as vehicles, due to bandwidth limitations and the inability of existing gateways to handle the data size, particularly with sensors like Lidar producing multiple terabytes of data daily.
Innovation Solution
A device in a serial network de-multiplexes traffic streams, quantizes, and applies data-type-specific compression to segregate and transform data before sending it as Internet Protocol (IP) traffic, optimizing data flow and reducing bandwidth requirements through a model-based, programmable compression mechanism.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is transmitted over existing network infrastructure (LTE gateway), then network compatibility is maintained, but bandwidth is insufficient to handle large data volumes
Solution Approach 1:
The system performs preliminary compression and quantization of sensing data before transmission. By pre-processing the data to reduce its size, the system prepares the data in advance to fit within the bandwidth constraints of existing LTE network infrastructure, resolving the contradiction between maintaining network compatibility and increasing data transmission capacity.
Solution Approach 2:
The system changes the parameter of data representation by applying quantization and compression algorithms. This transforms the data from a high-volume raw format to a compressed format with reduced bit depth, enabling more data to be transmitted over the same network bandwidth while maintaining compatibility with existing infrastructure.
2Quantity of substance
If data compression is applied, then bandwidth requirements are reduced, but data processing complexity increases
Solution Approach 1:
The system segments the data processing by separating compression operations at the edge device from decompression at the cloud platform. This division allows complex compression algorithms to be applied locally where computing resources are available, while the cloud platform handles decompression, thereby managing processing complexity across distributed system components.
Solution Approach 2:
Compression is performed as a preliminary action at the edge device before data leaves the vehicle. By completing the computationally intensive compression operation in advance, the system reduces the bandwidth requirements for transmission without requiring complex real-time processing during data transfer.
3Reliability
If all sensing data is transmitted to cloud, then data availability is maximized, but network bandwidth is overwhelmed
Solution Approach 1:
The system extracts only the essential features and compressed representations of sensing data for transmission to the cloud, rather than transmitting all raw data. This extraction approach maintains data availability for cloud-based analytics while significantly reducing network bandwidth consumption by leaving redundant and non-essential data at the edge device.
Solution Approach 2:
The system changes the data parameters through quantization and compression, transforming full-resolution sensing data into a compact representation that preserves essential information. This parameter transformation enables more efficient use of network bandwidth while maintaining sufficient data quality for cloud-based processing and analytics.
Data Source
AI summary
In one embodiment, a device in a serial network de-multiplexes a stream of traffic in the serial network into a plurality of data streams. A particular one of the data streams is associated with a particular endpoint in the serial network. The device determines that data from the particular data stream associated with the particular endpoint should be reported to an entity external to the serial network based on an event indicated by the data from the particular data stream. The device quantizes the data from the particular data stream. The device applies compression to the quantized data to form a compressed representation of the particular data stream. The applied compression is selected based on a data type associated with the data. The device sends a compressed representation of the particular data stream to the external entity as Internet Protocol (IP) traffic.