Dynamic Hash Partition Management for Database Systems

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing database management systems face challenges in dynamically managing hash partitions, particularly when changing the number of partitions, as it often requires data movement or complex metadata management, leading to inefficiencies and asymmetric partitions.

Innovation Solution

A method for dynamic hash partition management that associates partitions with partition numbers, selects an active partition number based on the current scheme, generates a partition identifier using a record key and active partition number, and inserts or retrieves records without moving existing data, allowing for seamless changes in partition schemes without redistributing data.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If data movement is used to conform to new partition scheme, then partition scheme can be changed, but large amounts of data must be moved which reduces efficiency

Engineering Contradiction:

Improvepartition scheme flexibilityVSAvoiddata movement cost

Core Design Contradiction:

Adaptability or versatilityVSLoss of energy

Solution Approach 1:

The patent implements dynamic partition management by allowing the number of partitions to change over time without requiring data movement. The system dynamically adjusts partition schemes while maintaining data integrity through a hash partitioning mechanism that redistributes data automatically based on the new partition count, eliminating the need for manual data migration.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent changes the partition scheme by modifying the partition count parameter while keeping the hash function and data storage structure unchanged. This allows the system to adapt to different partition requirements by simply changing the number of partitions parameter, avoiding costly data movement operations.

Inventive Principle:

Principle #35Parameter changes

2Adaptability or versatility

If splitting and merging partitions is used to change partition number, then partition scheme can be adjusted, but complicated metadata bookkeeping is required and asymmetric partitions may result

Engineering Contradiction:

Improvepartition number adjustabilityVSAvoidmetadata management complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system performs self-service partition management where the hash partitioning mechanism automatically redistributes data based on the new partition count without requiring manual intervention or complex metadata updates. The automatic redistribution process eliminates the need for complicated metadata bookkeeping while ensuring balanced partitions.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent replaces manual metadata management with an automated hash-based redistribution mechanism. Instead of manually tracking and updating metadata for partition splits and merges, the system uses a hash function to automatically determine data placement, simplifying the overall system complexity.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Adaptability or versatility

If existing data is redistributed to conform to new partition scheme, then partition scheme can be changed, but system performance decreases due to data movement

Engineering Contradiction:

Improvepartition scheme flexibilityVSAvoidsystem performance

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The patent performs preliminary action by pre-calculating the new partition identifiers based on the hash function and record keys before actual data placement. This allows the system to prepare the partitioning structure in advance, avoiding data movement during critical operations and maintaining high system performance while enabling flexible partition scheme changes.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS10685031B2Dynamic hash partitioning for large-scale database management systems

Publication Date: 2020.06.16 NEW RELIC INC

AI summary

A dynamic hash partition management scheme includes associating first partitions of a database with a first partition number, and second partitions of the database with a second partition number, associating first partitions of a database with a first partition number defining a total number of first partitions, and second partitions of the database with a second partition number defining a total number of second partitions. The method further includes, after receiving a request to insert a record in the database, determining an active partition number, and determining the first partition number as an active partition number. The method further includes determining a record key for the record, generating a partition identifier defining a partition of the first partitions using the record key and the active partition number, and inserting the record in the partition defined by the partition identifier.