Time-Series Database Execution for Real-Time Custom Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional database systems experience delays and inefficiencies in retrieving and analyzing large quantities of time-series data due to network limitations and the inability to perform custom, user-defined computational operations in real-time, leading to noticeable delays in data display.
Innovation Solution
A database system that includes memory storage, disk storage, and processors to store immutable time-series data, allowing for real-time execution of custom computational expressions and operations directly on the database, reducing the need for server analysis and improving data retrieval and processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in conventional database systems, then data storage capacity increases, but data retrieval time increases
Solution Approach 1:
The database system divides time-series data into discrete time intervals or buckets, organizing data chronologically into manageable segments. This segmentation allows the system to retrieve only relevant time-range data rather than scanning entire datasets, significantly reducing retrieval time while maintaining large storage capacity
Solution Approach 2:
The system introduces time as an explicit dimensional index for data organization and retrieval. By creating time-based indexes and using time-range queries, the system can efficiently locate and retrieve data within specific time windows without scanning all stored data, resolving the contradiction between storing large quantities of data and retrieving it quickly
2Adaptability or versatility
If custom computational expressions are executed on large datasets, then analysis capability improves, but processing time increases
Solution Approach 1:
The system pre-computes and stores aggregated statistics (such as sums, averages, minimums, maximums) for time-series data segments. When custom computational expressions are requested, the system can leverage these pre-computed values as a foundation, reducing the computational burden and processing time while maintaining the ability to execute complex analyses
3Reliability
If data files are updated frequently, then data freshness improves, but disk access overhead increases
Solution Approach 1:
The system implements periodic batching of data writes, where multiple data points are accumulated in memory and written to disk together in batch operations. This periodic writing approach reduces the frequency of disk access operations while ensuring data is persisted at regular intervals, balancing data freshness with reduced disk overhead
Solution Approach 2:
The system maintains in-memory copies of recently accessed or frequently queried data alongside the persistent disk storage. This caching mechanism allows the system to serve recent data from fast memory without requiring disk access, while periodic synchronization ensures disk storage remains updated with the latest data
Data Source
AI summary
A database system is described that includes components for storing time-series data and executing custom, user-defined computational expressions in substantially real-time such that the results can be provided to a user device for display in an interactive user interface. For example, the database system may process stored time-series data in response to requests from a user device. The request may include a start time, an end time, a period, and/or a computational expression. The database system may retrieve the time-series data identified by the computational expression and, for each period, perform the arithmetic operation(s) identified by the computational expression on data values corresponding to times within the start time and the end time. Once all new data values have been generated, the database system may transmit the new data values to the user device for display in the interactive user interface.