IoT Data Aggregation Server with On-Demand Transformation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The growth of IoT devices generating networked data poses challenges in data aggregation, transformation, and visualization, particularly due to the susceptibility of these devices to producing erroneous data, which can corrupt otherwise accurate datasets.
Innovation Solution
A server system that receives and stores raw IoT broadcast reports, applying pre-defined data transformations only upon request to produce transformed data, while preserving raw data for archival purposes, thereby eliminating erroneous data and reducing computational and memory load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored in raw format to identify potentially faulty electronic devices, then data integrity is improved, but computational load and memory requirements increase
Solution Approach 1:
The system performs preliminary data filtering and validation at the time of data ingestion, creating a cleaned subset of data that is stored separately from the raw data. This preliminary action eliminates erroneous data points before they can contribute to computational complexity in later analysis, while preserving raw data integrity for reference.
Solution Approach 2:
The data storage system is segmented into multiple layers: raw data storage, filtered/cleaned data storage, and processed data storage. This segmentation allows the system to maintain raw data for integrity purposes while creating separate, optimized datasets for computational analysis, thereby reducing the computational load associated with processing raw erroneous data.
2Reliability
If data is stored in raw format to identify potentially faulty electronic devices, then data integrity is improved, but memory requirements increase
Solution Approach 1:
Erroneous data points are extracted and removed from the dataset through automated validation rules and filtering algorithms. The cleaned data is then stored in an optimized format that requires less memory than storing all raw data, while the raw data is retained in a compressed or sampled manner for reference purposes only.
Solution Approach 2:
The system changes the state of data from raw unprocessed form to processed/cleaned form through transformation operations. This parameter change reduces the volume of data requiring storage by eliminating redundant and erroneous entries, thereby reducing memory requirements while maintaining data integrity through selective preservation of valid data points.
3Measurement precision
If periodic erroneous data is filtered to protect accurate datasets, then data accuracy is improved, but processing time increases
Solution Approach 1:
Data filtering and validation operations are performed preliminarily at the time of data ingestion rather than during analysis. Erroneous data points are identified and removed through pre-defined validation rules, transforming the data into a cleaned state before storage. This preliminary action significantly reduces processing time during subsequent analysis operations.
4Loss of information
If transformed data is generated on-demand from raw data, then data freshness is improved, but computational load increases
Solution Approach 1:
The system performs preliminary data transformation and cleaning operations when data is first ingested, creating a pre-processed version that is immediately available for analysis. This preliminary transformation eliminates the need for repeated heavy computational operations when generating transformed data, as the cleaned dataset is already prepared and stored for rapid retrieval.
Data Source
AI summary
A server has a processor and a memory connected to the processor. The memory stores instructions executed by the processor to receive broadcast reports from a network, where each broadcast report includes an account identifier, a group identifier, a data stream identifier and a value. Each broadcast report is stored in a database as raw data. A request for information in the database is received. In response to the request, the raw data is altered in accordance with a pre-defined data transformation to produce transformed data. A first visualization of the transformed data associated with a first data stream identifier of a first group identifier is rendered. A second visualization of the transformed data associated with a second data stream identifier of the first group identifier is rendered. The first visualization is supplied in conjunction with the second visualization in response to the request.