Message Broker Tiered Storage for Low-Latency Building Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing web services platforms face delays in data presentation due to the need for applications to retrieve and process raw timeseries data from relational databases, which can lead to significant latency in data visualization and analysis.
Innovation Solution
The implementation of a system with isolated compute environments, where a message broker processes and stores timeseries data in two tiers of storage: a fast response tier (such as RAM or solid-state drives) for immediate data access and a secondary tier (like magnetic disk drives) for long-term storage, allowing for efficient data retrieval and processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If raw timeseries data is stored in a relational database without significant organization or processing, then data storage is simplified, but data retrieval and processing time is significantly delayed
Solution Approach 1:
The patent pre-processes and organizes timeseries data into structured formats (e.g., time buckets, aggregated statistics) at the time of data ingestion rather than waiting until query time. This preliminary organization enables fast retrieval without sacrificing storage simplicity, directly resolving the contradiction between storage organization complexity and data retrieval time
Solution Approach 2:
The patent segments timeseries data into different time buckets and organizational structures (e.g., daily, weekly, monthly aggregations) that can be independently stored and retrieved. This segmentation allows the system to provide both simplified storage and fast retrieval by organizing data in pre-computed time-based segments
2Speed
If data is processed and presented in real-time, then data presentation latency is reduced, but system complexity increases
Solution Approach 1:
The system performs preliminary processing of timeseries data into various aggregation levels and formats during data ingestion. This pre-computation enables real-time data presentation without requiring complex processing at query time, as the data is already organized in ready-to-present formats
Solution Approach 2:
The patent introduces an intermediary data layer between the relational database and the application layer. This intermediary layer pre-organizes data into structured formats and provides optimized query interfaces, reducing the complexity burden on both the database and application layers while enabling real-time presentation
3Speed
If all timeseries data is kept in fast storage, then data access speed is improved, but storage cost and resource consumption increase
Solution Approach 1:
The patent applies different storage strategies to different portions of timeseries data based on their access patterns. Frequently accessed recent data is stored in fast storage, while historical data is archived in slower, cheaper storage. This local quality differentiation optimizes both access speed and resource consumption by matching storage media to data usage characteristics
Data Source
AI summary
A brokering system used in distributed building controls provides multiple tiers of storage. Data can be requested at any time during the processing of new data received from a building device. Complex event processors, low latency control, and/or display interfaces may request and be provided data from memory (e.g., tier one storage) of a message broker. Digital twins and/or optimization algorithms may wait until data is stored to local disk before information is obtained. Other applications may wait until data is saved in the remote database before using the data to perform calculations. Data may be compressed and stored in timeseries that are accessed by a single requested to the message broker.