Database Management System for High-Volume IoT Time Series Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-volume data systems, particularly those in IoT networks, face challenges in efficiently ingesting, storing, and retrieving large volumes of time series data from multiple sources while maintaining scalability and performance, especially in handling non-chronological data ingestion and providing fast query capabilities.
Innovation Solution
A database management system that includes an ingest container for processing and dividing time series data into sub-elements, generating statistical data, and storing them in a columnar format within an archive, cooked, and index container, allowing for parallel updates and out-of-order ingestion, with a query API for efficient data retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored in a traditional relational database format, then data integrity and query accuracy are maintained, but system scalability and ingestion speed deteriorate when handling high-volume IoT time series data
Solution Approach 1:
The system segments data storage into three distinct containers: archive container for raw time series data, cooked container for processed data, and index container for metadata. This segmentation allows each container to be optimized for its specific function, enabling high-volume data ingestion while maintaining query efficiency and simplifying individual component complexity.
Solution Approach 2:
The patent transitions from traditional relational database storage to a cloud-native containerized architecture using append blobs and data lakes. This dimensional change in storage approach enables parallel data ingestion and processing, dramatically improving productivity for high-volume IoT data while distributing system complexity across managed cloud services.
2Reliability
If data is processed and stored in real-time chronological order, then data consistency is maintained, but system performance and query speed deteriorate when handling non-chronological data streams
Solution Approach 1:
The system performs preliminary actions by generating statistical aggregates (min, max, mean, count) and creating index metadata during the data ingestion phase. This preliminary processing allows the index container to provide fast query performance for non-chronological data streams while the archive container maintains raw data consistency, resolving the contradiction between reliability and speed.
3Loss of information
If all raw time series data is retained and stored, then complete data availability is ensured, but storage costs and retrieval time increase
Solution Approach 1:
The system extracts essential statistical information (min, max, mean, count) and metadata from raw time series data during ingestion, storing these extracted elements in the cooked and index containers. This extraction ensures complete data availability for analysis while enabling fast retrieval of aggregated statistics without requiring access to all raw data, thus reducing retrieval time while preventing information loss.
Solution Approach 2:
Different containers provide different qualities of data: the archive container stores complete raw data for full data availability, while the index container provides pre-computed statistical aggregates for fast retrieval. This local quality differentiation allows the system to optimize for both data availability and retrieval time depending on the specific query requirements.
4Adaptability or versatility
If multiple data sources are ingested simultaneously, then system scalability is improved, but data processing complexity and coordination overhead increase
Solution Approach 1:
The ingest container serves as a universal entry point that handles data from multiple IoT sources simultaneously using a standardized interface. This multi-functional design allows diverse data sources to be ingested through a single container, improving adaptability while the underlying parallel processing architecture handles coordination automatically, reducing processing complexity.
Data Source
AI summary
A system includes computer-readable media having memory, one or more processors, and instructions in the memory that, when executed by the one or more processors, cause the processor(s) to: generate an archive container, a cooked container, an ingest container, and an index container; receive, in the ingest container, a plurality of time series data elements; identify a data format for each of the time series data elements; divide the plurality of time series data elements into a plurality of data sub-elements based on a corresponding append binary large objects (blobs) contained in the cooked container; generate, statistical data for the plurality of time series data elements for each of one or more index blobs contained in the index container; and output: the statistical data to the index container; the plurality of data sub-elements to the cooked container, and the plurality of time series data elements to the archive container.