MQTT Data Logger and Processor for Near Real Time Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing MQTT brokers do not provide a mechanism to save messages sent across the M2M network protocol, necessitating manual data collection, parsing, cleaning, and sorting for processing by Modeling and Simulation (M&S) tool suites.
Innovation Solution
A Digital Mission Engineering (DME) MQ Telemetry Transport (MQTT) Data Logger and Processor is developed to subscribe to an MQTT broker, log all messages, clean and format the data, and convert it into a format ingestible by M&S tool suites, enabling near real-time data analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If known MQTT brokers are used, then message routing and client management are provided, but message storage and logging capabilities are not provided
Solution Approach 1:
The patent combines message storage and logging functionality with the MQTT broker by implementing a data logger module that integrates directly with the broker's message handling mechanisms. This allows messages to be both routed to subscribers and simultaneously stored in a database without requiring separate storage infrastructure.
Solution Approach 2:
The MQTT broker is enhanced to perform multiple functions: traditional message routing, client connection management, and now also message persistence and logging. The unified broker handles both real-time message delivery and long-term data retention through a single integrated system.
2Productivity
If manual data collection and processing is performed, then data can be processed by M&S tool suites, but significant manual effort and time are required
Solution Approach 1:
The system performs preliminary data processing by automatically collecting, parsing, cleaning, and formatting messages as they are received from the MQTT broker. This pre-processing transforms raw messages into M&S-ready formats before analysis begins, eliminating the need for manual data preparation steps.
Solution Approach 2:
The data logger and processor operate autonomously to handle the complete data pipeline from message reception to formatted output. The system automatically manages data collection, parsing, cleaning, and conversion without requiring manual intervention, enabling researchers to focus on analysis rather than data preparation.
3Speed
If MQTT messages are logged and processed automatically, then near real-time data analysis is enabled, but additional processing infrastructure is required
Solution Approach 1:
The patent merges the data logging, processing, and formatting functions into a single integrated module that works closely with the MQTT broker. This unified architecture reduces infrastructure complexity by combining multiple processing steps into one cohesive system rather than requiring separate components for each function.
Data Source
AI summary
Disclosed is an MQ Telemetry Transport (MQTT) Data Logger and Processor for Near Real Time Data Analysis that is configured to subscribe to a Machine-to-Machine (M2M) network protocol and store all the messages sent to an MQTT broker. Once all of the messages have been logged, the presently disclosed invention further features cleans and places the data into a format that is usable or accessible by modeling and simulation (M&S) tool suites, as one example. These aspects can be utilized by test and evaluation (T&E) engineers to rapidly evaluate system characteristics and performance. Moreover, the disclosed methodology can be used to augment technical data within M&S scenarios for verification and validation (V&V) and post processing.