Adaptive Time Series Database Schema for Dynamic Partitioning

Solution Overview

Problem

Conventional relational databases face challenges in managing large quantities of time-sensitive data, particularly in dynamic environments, due to limitations in scalability and fixed partitioning schemes, which impact query response times and efficiency.

Innovation Solution

The development of an adaptive time series database schema that dynamically adjusts based on the data shape and query patterns, allowing for variable partitioning and self-tuning to optimize storage and retrieval processes.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Device complexity

If conventional relational databases use fixed partitioning schemes, then data storage is simplified, but query response time is significantly impacted and scalability is limited

Engineering Contradiction:

Improvedata storage simplicityVSAvoidquery response time

Core Design Contradiction:

Device complexityVSSpeed

Solution Approach 1:

The patent implements dynamic partitioning schemes that automatically adjust based on query patterns and data characteristics. The system monitors query workloads and re-partitions data accordingly, transforming the static partitioning structure into a dynamic one that adapts to changing requirements, thereby improving query response time without sacrificing storage simplicity

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system changes partitioning parameters (such as partition keys, partition counts, and distribution strategies) based on analyzed query patterns. By modifying these parameters dynamically, the database optimizes query performance for different workload types while maintaining manageable storage complexity

Inventive Principle:

Principle #35Parameter changes

2Quantity of substance

If relational databases manage large quantities of time-sensitive data, then data comprehensiveness is improved, but scalability and management efficiency deteriorate

Engineering Contradiction:

Improvedata quantityVSAvoidmanagement efficiency

Core Design Contradiction:

Quantity of substanceVSProductivity

Solution Approach 1:

The patent segments time-sensitive data into specialized time-series partitions with optimized storage formats. By dividing the large data volume into manageable, purpose-optimized segments, the system maintains comprehensive data storage while improving management efficiency through targeted query processing and reduced scan requirements

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system introduces an intermediary layer between the relational database and time-series data, implementing a specialized time-series storage format and query optimization layer. This intermediary handles the complexity of large-scale time-sensitive data management, allowing the underlying relational database to maintain simplicity while achieving high scalability and efficiency

Inventive Principle:

Principle #24Intermediary (Mediator)

3Device complexity

If fixed partitioning schemes are used in databases, then storage structure is simplified, but query efficiency and throughput are reduced

Engineering Contradiction:

Improvestorage structure simplicityVSAvoidquery throughput

Core Design Contradiction:

Device complexityVSProductivity

Solution Approach 1:

The patent implements dynamic partitioning schemes that automatically adjust based on query patterns and data characteristics. The system monitors query workloads and re-partitions data accordingly, transforming the static partitioning structure into a dynamic one that adapts to changing requirements, thereby improving query response time without sacrificing storage simplicity

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system changes partitioning parameters (such as partition keys, partition counts, and distribution strategies) based on analyzed query patterns. By modifying these parameters dynamically, the database optimizes query performance for different workload types while maintaining manageable storage complexity

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS11500829B2Adapting time series database schema

Publication Date: 2022.11.15 VMWARE INC

AI summary

In a computer-implemented method for adapting time series database schema of a time series database, time series data ingested into a time series database according to a time series database schema is accessed over a time period, wherein time series data comprises a plurality of dimensions. The time series data of the time period is analyzed to determine a data shape of the time series data of the time period. It is determined whether to adapt the time series database schema based at least in part on the data shape of the time series data of the time period. In some embodiments, the time series database schema is adapted based at least in part on the data shape of the time series data of the time period. Time series data is then ingested into the time series database according to the adapted time series database schema.