Separating Metadata from Time Series Data in Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional techniques for processing metadata are resource-intensive and cause delays due to the need to update millions of metric time series (MTS) objects whenever a single dimension is updated, as MTS objects are de-normalized and include all properties and information from associated dimensions.
Innovation Solution
The solution involves storing metadata and MTS objects separately in a distributed system using in-memory data structures. This allows updates to metadata objects to remain with the metadata without propagating to MTS objects, and enables a logical in-memory join to associate metadata with the appropriate MTS object during queries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If MTS objects are stored with all properties and information from associated dimensions (de-normalized storage), then query performance is improved, but update complexity and resource consumption increase significantly
Solution Approach 1:
The patent segments the storage of MTS objects and metadata into separate in-memory data structures. MTS objects are stored in one data structure while metadata objects are stored in another, allowing independent updates to metadata without affecting MTS objects. This segmentation resolves the contradiction by maintaining query performance through efficient data retrieval while reducing update complexity through independent data structures.
Solution Approach 2:
The patent introduces in-memory data structures as intermediaries between metadata objects and MTS objects. These data structures act as mediators that correlate metadata with MTS objects through keys, enabling queries to join the data efficiently while preventing direct propagation of metadata updates to MTS objects. This intermediary approach maintains query performance while eliminating the need for expensive update propagation.
2Reliability
If MTS objects are updated whenever a dimension is updated, then data consistency is maintained, but resource consumption and processing time increase significantly
Solution Approach 1:
The patent segments updates to metadata objects from updates to MTS objects by storing them in separate in-memory data structures. When metadata is updated, only the metadata data structure is modified, not the MTS objects. This segmentation maintains data consistency through proper joins during queries while significantly reducing resource consumption by eliminating unnecessary update propagation.
Solution Approach 2:
The patent uses in-memory data structures to store correlations between metadata and MTS objects as reference copies rather than full data copies. These data structures store keys and references that enable efficient joining during queries without requiring actual data duplication. This copying approach maintains data consistency relationships while minimizing resource consumption compared to full data propagation.
3Ease of operation
If all properties from dimensions are included in each MTS object, then data accessibility is improved, but storage efficiency decreases due to redundancy
Solution Approach 1:
The patent segments data into separate in-memory data structures for MTS objects and metadata objects, storing each type of data only once. This segmentation eliminates redundancy while maintaining accessibility through efficient joining mechanisms. MTS objects store only their core data, and metadata is stored separately, reducing the quantity of data stored while preserving ease of operation through correlated queries.
Solution Approach 2:
The patent introduces in-memory data structures as intermediaries that efficiently correlate MTS objects with metadata without storing redundant data. These intermediaries use keys and references to maintain relationships between data types, enabling efficient data accessibility while improving storage efficiency by avoiding redundant data storage across multiple objects.
Data Source
AI summary
According to embodiments, a data stream including a plurality of time series data is received and metadata objects are extracted from the data stream. The metadata objects are associated with metrics time series (MTS) objects. The metadata objects and MTS objects are stored via separate in-memory data structures in a logical database. The in-memory data structures include information that correlates the metadata objects with the MTS objects. Any updates to the metadata objects will stay with the metadata objects and do not propagate to the MTS objects. A logical in-memory join may be performed to associate the metadata objects with the appropriate MTS object according to the in-memory data structures when a query for an MTS object is received.