IoT Data Compaction Using Attribute-Based Representative Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing volume of data from IoT devices poses economic and technical challenges in provisioning storage capacity and bandwidth, as existing compression methods require significant storage capacity and may not be suitable for devices with limited resources.
Innovation Solution
A data compaction system that dynamically selects compaction schemes based on attributes like bounded deviation, threshold sensitivity, and skew, using methods such as function fitting, piecewise fitting, and bitmap indexing to represent data efficiently, allowing edge devices to retain more data without external storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If compression is used to reduce data volume, then storage capacity requirements are reduced, but appreciable storage capacity is still required and bandwidth provisioning remains challenging
Solution Approach 1:
The patent extracts only the essential characteristics and patterns from the original data streams, storing merely the representative features rather than the complete data. This is achieved through schema-based abstraction where only relevant data elements are retained in the time-series database, eliminating redundant information while preserving analytical value.
Solution Approach 2:
Instead of compressing data to fit storage constraints, the patent inverts the approach by pre-defining data schemas that capture only the necessary data elements from the beginning. This schema-driven method filters data at ingestion rather than compressing it later, fundamentally changing the data storage paradigm from volume-based to value-based storage.
2Productivity
If data is streamed at high rates from multiple devices, then data processing capability is improved, but bandwidth provisioning and storage capacity become severe economic and technical hurdles
Solution Approach 1:
The system extracts only the meaningful signals from high-volume data streams by applying predefined schemas that identify and retain only relevant data elements. This extraction process occurs at the edge devices and gateway, filtering out redundant information before transmission, thereby maintaining processing capability while reducing bandwidth and storage requirements.
Solution Approach 2:
The patent segments the data processing function across multiple levels: edge devices perform local filtering and schema application, gateways aggregate and further process data, and central systems handle analytical workloads. This segmentation distributes the processing burden and reduces the volume of data that must be transmitted and stored centrally.
3Measurement precision
If lossless compression is used to preserve original data values, then data accuracy is maintained, but appreciable storage capacity is still required
Solution Approach 1:
The patent extracts only the essential measurement values that are necessary for analysis, storing them with precise schema-defined data types. By focusing on extracting only the meaningful measurement data rather than preserving all original data representations, the system maintains measurement precision while significantly reducing storage capacity requirements.
4Quantity of substance
If lossy compression is used to reduce storage requirements, then storage capacity is reduced, but data may be discarded that could be important
Solution Approach 1:
The patent applies different data retention strategies to different data elements based on their local quality and importance characteristics. Schema definitions specify which data elements require precise retention and which can be aggregated or filtered, allowing the system to maintain reliability for critical measurements while reducing storage for less important data.
Solution Approach 2:
The system changes parameters such as data sampling rates, aggregation intervals, and precision levels based on the specific characteristics of each data stream and its importance. This allows dynamic adjustment of storage requirements while maintaining reliability for critical data through schema-defined retention policies.
Data Source
AI summary
Methods, apparatuses, and storage media associated with compaction of data from one or more computing devices are disclosed. In various embodiments, one or more Internet of Things (IoT) devices may transmit information to a computing system. The computing system may group together raw data received from these one or more IoT devices based on a shared attribute. The computing system may select a compaction scheme to represent the knowledge conveyed by a group of the raw data. The computing system may apply this compaction scheme to the group of raw data to generate data that is representative of the group of raw data. Other embodiments may be disclosed or claimed.