Time Series Database Metadata Filtering for Analysis Speed
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face challenges in efficiently processing and analyzing large volumes of time series data, leading to slow operation speeds and impractical analysis times due to the need to read and process entire datasets.
Innovation Solution
A system comprising a communications interface for receiving time series data from multiple sensors, one or more storage devices for storing databases containing the time series data and metadata, and one or more processors configured to query the metadata database, access relevant time series data, and perform computations on the accessed data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the entire stored time series data is read for each analysis operation, then complete data analysis can be performed, but the operation speed becomes slow and analysis time increases significantly
Solution Approach 1:
The patent extracts and stores metadata (summary information, statistics, and characteristics) separately from the full time series data. This metadata is pre-computed and stored in a metadata database, allowing the system to query and filter data without reading the entire dataset. When analysis is needed, only relevant portions of the data are retrieved based on metadata filtering criteria, dramatically reducing analysis time while maintaining complete analysis capability for the selected subset.
Solution Approach 2:
The system performs preliminary actions by pre-computing and storing metadata (such as min/max values, averages, data ranges, and other summary statistics) before actual analysis operations. This preliminary processing allows the system to quickly determine which data portions are relevant to current analysis needs, avoiding the need to read and process entire datasets for each operation.
2Productivity
If metadata database is queried to filter data series, then data access time is reduced, but system complexity increases due to dual database structure
Solution Approach 1:
The patent segments the database system into two distinct components: a metadata database storing summary information and a data database storing full time series data. This segmentation allows independent optimization of each component - the metadata database can be small and highly optimized for query operations, while the data database can be structured for efficient data retrieval. The segmentation resolves the complexity issue by creating two simple, specialized databases rather than one complex database attempting to handle both metadata and full data.
Solution Approach 2:
The metadata acts as an intermediary layer between the query system and the full time series data. Instead of directly querying and filtering large datasets, the system first queries the metadata database to identify relevant data portions, then uses this information to efficiently retrieve only necessary data from the data database. This intermediary metadata layer simplifies the overall system architecture by providing a filtering mechanism that reduces the burden on the data database.
Data Source
AI summary
A system includes a communications interface configured to receive a plurality of data series captured by a plurality of sensors. A first database includes at least a first data series of the plurality of data series. A second database includes metadata related to the first data series. One or more processors are configured to receive an instruction including an indication of a computation to perform and metadata filter criteria, query the second database to determine at least a portion of the first data series that satisfies the metadata filter criteria by reference to at least the metadata related to the first data series, access, from the first database and based on results of the query of the second database, the portion of the first data series, and perform the computation on the portion of first data series accessed from the first database.