Message Broker Architecture for Timeseries Compression and Fast Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing web services platforms delay data presentation by storing raw timeseries data without significant organization or processing, leading to inefficient retrieval and presentation of timeseries data.
Innovation Solution
A method involving quantization, integer encoding, and compression of timeseries data, with differential encoding and bit packing, allowing for faster access and storage in multiple tiers of storage, and utilizing message brokers for enhanced data handling and processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If raw timeseries data is stored without significant organization or processing, then data storage is simple, but data retrieval and presentation are delayed
Solution Approach 1:
The patent applies preliminary action by organizing and processing timeseries data before storage. The system performs quantization, differential encoding, and compression on incoming data streams, transforming raw data into a structured format with reduced redundancy. This pre-processing enables significantly faster retrieval speeds when applications request data, as the data is already organized in an efficient format rather than requiring processing at query time.
2Speed
If data is compressed and organized, then data retrieval speed improves, but data storage complexity increases
Solution Approach 1:
The patent applies parameter changes by transforming the data representation parameters. The system quantizes continuous values to integer representations, applies differential encoding to capture only changes between adjacent values, and uses bit packing to optimize storage density. These parameter transformations reduce the storage footprint while enabling faster access, as the compressed format requires less I/O operations and can be loaded more efficiently into memory.
Solution Approach 2:
The patent applies segmentation by dividing the timeseries data into manageable chunks or segments that can be processed and stored independently. The system organizes data by device, by time ranges, and by data types, creating a segmented storage structure that allows applications to retrieve only the specific segments they need rather than processing entire datasets. This segmentation reduces the complexity of data management while improving access speed for common query patterns.
3Productivity
If multiple processing steps are performed on timeseries data, then data presentation efficiency improves, but processing time increases
Solution Approach 1:
The patent applies preliminary action by performing quantization, differential encoding, and compression on data immediately upon ingestion, before the data needs to be presented to applications. This upfront processing transforms raw timeseries data into a compact, organized format that requires minimal additional processing during retrieval. The result is that data presentation efficiency improves significantly, as the bulk of the processing work has already been completed, and applications can access pre-processed data with minimal delay.
Data Source
AI summary
One or more message brokers configured to communicate over a message bus and provide one or more features and the one or more features are executed in response to a field in a first message communicated to a message broker of the one or more message brokers. The operations also include creating a second set of the one or more message brokers in a second set of isolated compute environments. The second set of the one or more message brokers capable of performing an additional feature in addition to the one or more features. The communications over the message bus include sending a structured data format. The second set of one or more message brokers are configured to execute extension instructions related to the additional feature in response to a second message containing a specified field in the structured data format. The second set of one or more message brokers are deployed to the message bus while the first set of one or more message brokers communicates over the message bus.