Versioned Metadata Management for Time-Series Database Query Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed time-series databases face challenges in efficiently querying and managing data across multiple storage tiers due to differences in storage formats and access characteristics, leading to increased latency and complexity in data retrieval and storage.
Innovation Solution
The implementation of adaptive querying techniques that consult a metadata service to identify relevant partitions and storage tiers, converting data into a common format to create a unified logical representation, and using versioned metadata management to ensure query correctness and durability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is stored across multiple storage tiers with different formats and access characteristics, then storage capacity and data management flexibility are improved, but query latency and system complexity increase
Solution Approach 1:
The patent introduces a metadata service as an intermediary layer between the query interface and multiple storage tiers. This metadata service maintains versioned metadata that maps logical data identifiers to physical storage locations across different tiers (hot, warm, cold storage). When a query arrives, the system consults the metadata service to determine the optimal storage tier and format, eliminating the need for queries to search through all storage tiers sequentially. This resolves the contradiction by providing storage flexibility through multiple tiers while maintaining low query latency through intelligent routing via the metadata intermediary.
Solution Approach 2:
The system performs preliminary actions by pre-computing and storing metadata that describes the location, format, and accessibility of data across different storage tiers. This metadata is maintained in advance and updated whenever data is moved or transformed between tiers. When queries need to access data, the pre-computed metadata is already available to guide efficient retrieval, avoiding the time penalty of real-time analysis of storage tier characteristics. This preliminary preparation resolves the contradiction between storage versatility and query speed.
2Reliability
If data is converted to common format across storage tiers, then query correctness and data consistency are improved, but processing time and computational resources increase
Solution Approach 1:
The patent applies partial action by converting data to common format only when necessary for query correctness, rather than converting all data universally. The metadata service tracks the format of data in each storage tier and determines on-demand whether format conversion is needed for a given query. This selective approach maintains query correctness while minimizing unnecessary processing time and computational resource consumption, resolving the contradiction between reliability and processing efficiency.
3Reliability
If versioned metadata management is implemented, then query correctness and durability are improved, but metadata storage requirements and system complexity increase
Solution Approach 1:
The patent segments metadata management by organizing versioned metadata into structured formats that separate different aspects of data description (location, format, version, storage tier). This segmentation allows the system to manage complexity through modular metadata structures that can be independently maintained and queried. The segmented metadata approach maintains query correctness through version tracking while reducing overall system complexity through organized, hierarchical metadata structures.
Data Source
AI summary
Methods, systems, and computer-readable media for versioned metadata management for a time-series database are disclosed. A metadata service of a distributed time-series database stores, to one or more metadata storage resources, first metadata descriptive of a table. The first metadata is associated with a version number in a logical sequence for the table. The metadata service stores, in a log, data indicative of one or more metadata updates for the table. The metadata update(s) are associated with an additional version number higher than the version number in the logical sequence. Responsive to a read request, the metadata service determines a most recent version number that has been applied from the log to the metadata storage resource(s). Responsive to the read request and based (at least in part) on the most recent version number, the metadata service returns either the first metadata or second metadata comprising the update(s).