Intelligent Data Queue for Industrial Asset Time Series Ingestion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data ingestion pipelines in industrial environments face inefficiencies due to lack of concurrent updates and inaccurate indexing, leading to inefficient data processing and querying, which affects the real-time understanding of industrial asset status and performance.
Innovation Solution
Implementing an intelligent data queue with a two-layer cache system to manage data modification requests, allowing for concurrent processing and detection of overlapping operations, thereby optimizing data ingestion and ensuring accurate, real-time data contextualization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional data ingestion pipelines are used without concurrent update support, then data processing simplicity is maintained, but data processing efficiency and real-time asset status understanding deteriorate
Solution Approach 1:
The patent segments the data ingestion pipeline into multiple worker processes that can concurrently handle different data modification requests. Each worker process independently processes payloads from the event queue, enabling parallel data processing while maintaining system manageability through modular architecture
Solution Approach 2:
The patent introduces an event queue as an intermediary component between data sources and the processing system. This queue buffers incoming data modification payloads and distributes them to worker processes, decoupling the production and consumption of data while enabling concurrent processing without direct complexity in the data pipeline
2Measurement precision
If traditional indexing methods are used without overlap detection, then indexing simplicity is maintained, but measurement precision and data accuracy deteriorate
Solution Approach 1:
The patent implements preliminary overlap detection before committing data modifications to the time-series database. The system checks whether incoming data payloads overlap with existing data ranges using the overlap detection module, preventing inaccurate indexing by identifying conflicts before they occur, thereby ensuring measurement precision without requiring complex post-processing correction mechanisms
Data Source
AI summary
Various embodiments relate to managing a database to modify data related to industrial assets using an intelligent data queue for data ingestion. Embodiments are configured to receive a data modification payload associated with a request to modify a portion of data stored in a time series database. Embodiments are also configured to parse the data modification payload to identify a type of modification with respect to the data stored in the time series database. Embodiments are also configured to store the data modification payload in at least one of a first data cache configured for storing pending data modification payloads and a second data cache configured for processing data modification payloads. In response to a determination that the data modification payload is stored in the second data cache, the data stored in the time series database is modified based at least in part on the type of modification.