HTTP Event Custom Field Extraction for IoT Data Ingestion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Internet-connected devices often lack access to complex communication infrastructures for data storage and analysis, and they face challenges in processing, securing, and scaling large volumes of data due to limited processing capabilities and time constraints for developers.
Innovation Solution
A data intake and query system that processes and stores raw machine data as events, using a flexible schema to extract information, allowing for late-binding schema application during search time, and enabling efficient data retrieval and analysis across disparate data sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional data logging functionality is added to internet-connected devices, then data storage capability is improved, but device complexity and development time increase
Solution Approach 1:
The patent introduces a centralized data storage system that acts as an intermediary between internet-connected devices and data storage. Devices send data through HTTP requests to this system, which handles storage, processing, and retrieval. This eliminates the need for complex local storage infrastructure on each device while maintaining data storage capabilities.
Solution Approach 2:
The data storage system provides multiple functions including data storage, processing, retrieval, and analysis through a unified HTTP-based interface. This universal system serves diverse data types from various devices without requiring device-specific storage solutions, reducing overall system complexity.
2Quantity of substance
If data logging functionality is added to internet-connected devices, then data collection capability is improved, but processing capability requirements increase
Solution Approach 1:
The patent extracts data processing functionality from the devices themselves and relocates it to a centralized server system. Devices only need to send raw data via simple HTTP requests, while the server handles complex processing, analysis, and storage operations, significantly reducing processing capability requirements at the device level.
Solution Approach 2:
The centralized data storage system serves as an intermediary that assumes the processing burden. It receives raw data from devices, processes it using server-side resources, and returns processed results, allowing devices with limited processing power to still achieve advanced data analysis capabilities.
3Quantity of substance
If data from multiple internet-connected devices is stored, then data volume is improved, but scalability and security management become more difficult
Solution Approach 1:
The patent implements a standardized HTTP-based interface for all data submissions regardless of device type or data format. This homogeneous approach allows the system to scale to multiple devices without increasing complexity, as each device interacts with the system through the same uniform protocol and interface.
Solution Approach 2:
The centralized system provides universal data handling capabilities that work with diverse data types and devices through a single unified platform. This includes universal authentication, storage, processing, and retrieval functions that scale automatically with the number of connected devices without requiring device-specific configurations.
Data Source
AI summary
A data intake and query system receives a message including raw machine via an internet protocol (IP) such as the hypertext transfer protocol (HTTP). The message includes a distinct payload portion and a distinct custom field portion. The payload portion includes raw machine data, while the custom field portion includes values for fields. An event that includes the raw machine data and the values is generated from the payload portion and the values are extracted from the custom field portion. The event is then stored such that the values are associated with the event.