Time Series Data Stager for High-Throughput Ingestion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional time series databases face inefficiencies in data ingestion, particularly with sliding window-based data processing, leading to limited ingestion rates and delayed query availability, especially when handling large volumes of data from sensors like smart meters.
Innovation Solution
A time series data stager system that preprocesses and optimizes data ingestion by using a sliding window approach, leveraging in-core acceleration and ingestion optimization techniques to enhance throughput and minimize data loss, allowing for sustained high ingestion rates and rapid query availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional time series databases use traditional data ingestion methods, then data can be stored in the database, but the ingestion rate is limited and query availability is delayed
Solution Approach 1:
The patent applies preliminary action by pre-processing and staging data in memory before it is permanently written to the time series database. The stager component buffers incoming time series data, performs preliminary formatting and validation, and prepares data blocks for efficient bulk ingestion. This allows the system to accept data at high rates and then transfer it to the database at optimized intervals, eliminating the bottleneck of traditional row-by-row insertion and enabling both high ingestion rates and rapid query availability.
2Quantity of substance
If the system handles large volumes of time series data from sensors, then data coverage is comprehensive, but ingestion throughput is reduced
Solution Approach 1:
The patent applies segmentation by dividing the data ingestion process into distinct stages: data reception, staging in memory, formatting into data blocks, and bulk insertion into the database. The stager component segments incoming data streams into manageable chunks that can be efficiently processed and transferred. This segmentation allows the system to handle large volumes of data from multiple sensors simultaneously without overwhelming the database ingestion pipeline, maintaining high throughput while processing comprehensive data sets.
Solution Approach 2:
The patent introduces an intermediary component (the stager) between the data sources and the time series database. This stager acts as a buffer and preprocessing layer that decouples the high-rate data generation from the database ingestion process. It performs data validation, formatting, and temporary storage in memory, then transfers data in optimized batches to the database. This intermediary enables the system to handle large data volumes from numerous sensors while maintaining high ingestion throughput by preventing direct coupling between data sources and the database.
3Productivity
If data is ingested rapidly into the time series database, then ingestion throughput is high, but data loss may occur during power outages
Solution Approach 1:
The patent applies beforehand cushioning by implementing a stager component that buffers incoming time series data in memory before permanent storage in the database. This in-memory buffer acts as a cushion that protects against data loss during power outages. When data arrives, it is first staged in the buffer with sufficient capacity to hold a window of data points. If a power outage occurs during bulk insertion to the database, the stager can retain the staged data and resume insertion after recovery, preventing data loss while maintaining high ingestion throughput during normal operation.
Data Source
AI summary
A time series data stager that receives input data sets and outputs output data blocks for ingestion into a time series database, with the out data blocks being sent at timings according to a sliding window based on a predetermined time.