Per-Partition Hardware Scaling for Distributed Database Shards

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing database sharding systems waste computing and storage resources by uniformly assigning hardware capacity to all data partitions, leading to higher operating costs due to overprovisioning on partitions that do not require high capacity.

Innovation Solution

A system that dynamically determines and configures different hardware capacities for each data partition based on operational demands, allowing higher capacity for demanding partitions and lower capacity for less demanding ones, thereby optimizing resource utilization.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If uniform hardware capacity is assigned to all data partitions, then reliability is improved through consistent performance, but computing and storage resources are wasted due to overprovisioning on partitions that do not require high capacity

Engineering Contradiction:

Improveconsistent performanceVSAvoidcomputing and storage resource waste

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The patent applies local quality by assigning different hardware capacities to different data partitions based on their specific workload requirements. The system evaluates metrics such as storage capacity, query complexity, and access frequency for each partition, then configures appropriate hardware resources (CPU, memory, storage) for each partition individually rather than uniformly across all partitions. This ensures that each partition receives the precise level of resources it needs, eliminating overprovisioning while maintaining reliable performance where necessary.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The patent implements dynamics by making hardware capacity assignment adaptive and changeable over time. The system continuously monitors workload metrics and dynamically adjusts the hardware capacity allocated to each data partition based on changing demands. This allows the system to scale resources up or down for specific partitions as needed, rather than committing to fixed uniform capacity assignments, thereby optimizing resource utilization while maintaining performance reliability.

Inventive Principle:

Principle #15Dynamics

2Productivity

If higher hardware capacity is allocated to demanding partitions, then productivity is improved for those partitions, but device complexity increases due to heterogeneous configuration management

Engineering Contradiction:

Improvepartition performanceVSAvoidheterogeneous configuration management

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent applies segmentation by dividing the database system into independent data partitions that can be managed and configured separately. Each partition is treated as an independent unit with its own hardware capacity requirements, allowing the system to optimize performance for demanding partitions without affecting others. This segmentation enables selective resource allocation and simplifies the management of heterogeneity by isolating complex configurations to specific partitions rather than requiring system-wide complexity.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent utilizes parameter changes by dynamically adjusting hardware capacity parameters (CPU allocation, memory size, storage capacity, I/O performance) for each data partition based on measured workload characteristics. The system modifies these parameters in response to changing demands, allowing high-performance configurations for demanding partitions while maintaining simpler configurations for less demanding ones, thereby managing device complexity through controlled parameter variation rather than fixed homogeneous settings.

Inventive Principle:

Principle #35Parameter changes

3Loss of energy

If dynamic hardware capacity allocation is implemented, then resource utilization is optimized, but system complexity increases due to continuous monitoring and adjustment mechanisms

Engineering Contradiction:

Improveresource utilization efficiencyVSAvoidmonitoring and adjustment mechanisms

Core Design Contradiction:

Loss of energyVSDevice complexity

Solution Approach 1:

The patent implements feedback mechanisms by continuously monitoring workload metrics such as storage capacity usage, query complexity, access frequency, and hardware resource utilization for each data partition. Based on this feedback, the system automatically adjusts the hardware capacity allocated to each partition to match actual demands. This closed-loop feedback control optimizes resource utilization efficiency by ensuring resources are allocated according to real-time needs rather than static configurations, while managing system complexity through automated decision-making based on predefined metrics and thresholds.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS20250342175A1Techniques for dynamically scaling hardware capacity used to host data partitions of a database

Publication Date: 2025.11.06 MONGODB INC

AI summary

Some embodiments provide a system for optimizing the operational efficiency of a distributed database system configured to store data divided among a plurality of data partitions. The distributed database system comprises database hardware for hosting the plurality of data partitions. The system determines, for each of multiple data partitions, a hardware capacity for hosting the data partition. The system configures the database hardware based on hardware capacities determined for hosting the data partitions.