Cascading Polyglot Data Flow for Time-Series Database Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current time-series databases face challenges in efficiently managing the life cycle of time-series data, including ingestion, storage, and retrieval, especially in high-demanding scenarios, due to limitations in data models and scalability, leading to increased complexity and resource overhead.
Innovation Solution
The proposed method, Cascading Polyglot Data Flow, employs multiple interrelated data models along a data flow stream path, optimizing operations based on data aging and user requirements, allowing for efficient ingestion, storage, and retrieval of time-series data by cascading data through pools with different models, such as key-value, short-column, and long-column models, and utilizing polyglot abstraction layers for hybrid queries and on-demand data transformation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single data model is used for all time-series data operations, then the database structure is simple and easy to implement, but the efficiency of data ingestion, storage, and retrieval deteriorates in high-demanding scenarios
Solution Approach 1:
The patent segments the database into multiple data pools (first data pool, second data pool, third data pool) each employing different data models (key-value, short-column, long-column) optimized for specific operational phases. This segmentation allows each pool to handle particular data lifecycle stages efficiently, resolving the contradiction between structural simplicity and operational efficiency.
Solution Approach 2:
The patent implements dynamic data model selection based on data aging and operational requirements. Data automatically transitions between different data models as it ages, with the system adapting its storage and retrieval strategies accordingly. This dynamic approach enables the database to maintain optimal performance across varying workloads without requiring a complex static structure.
2Productivity
If multiple data models are employed along the data flow path, then the efficiency of data ingestion, storage, and retrieval is improved, but the device complexity and resource overhead increase
Solution Approach 1:
The patent applies preliminary action by pre-defining the cascade path and data model transitions before data ingestion begins. Each data pool is pre-configured with its specific data model and transformation rules, allowing data to flow automatically through the optimized path without requiring complex real-time decision-making. This reduces operational complexity while maintaining high efficiency.
Solution Approach 2:
The patent introduces intermediate data pools that act as mediators between different data models. These intermediate pools perform controlled transformations and cascading operations, simplifying the transition between different data models. The intermediary pools manage the complexity of multiple data models by providing structured, automated transition mechanisms rather than direct complex interactions.
3Device complexity
If data is stored in a single data pool with one data model, then the device complexity is low, but the adaptability to different use cases and data aging requirements is limited
Solution Approach 1:
The patent applies local quality by assigning different data models to different data pools based on their specific functional requirements. The first data pool uses key-value models for rapid ingestion, the second uses short-column models for intermediate storage, and the third uses long-column models for historical analysis. Each local segment (data pool) has optimized quality characteristics suited to its specific role in the data lifecycle.
Solution Approach 2:
The patent utilizes parameter changes by transforming data between different model parameters as it cascades through the pools. Data transitions from key-value parameters in the first pool to short-column parameters in the second pool, and finally to long-column parameters in the third pool. These parameter changes enable the system to adapt to different use cases and data aging requirements while maintaining a relatively simple overall structure.
Data Source
AI summary
A method is used for managing a flow of data in at least one database, wherein said database is configured with at least two data models of data storage. In said method, during a first period of time, a first data flow portion is received in a computer, and the first data flow portion is then stored in a first data pool of the database according to the first data model. Then, after the first period of time, a transformation is made on the first portion of the data and the transformed first data is assigned to a second data model, and the first data flow portion is then transferred from the first data pool to a second data pool. The process is repeated with at least a second data flow portion and can be extended to more transformations, and, thus, to more data models, and more data pools accordingly.