Flexible Data Partitioning via Dynamic Mapping Metadata

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data partitioning methods lack flexibility in managing the number of partitions, assigning data, and adjusting partitioning keys, making it difficult to optimize performance, reliability, and scalability.

Innovation Solution

A flexible partitioning scheme that allows users to dynamically add or remove partitions, control growth rates, and change mapping methodologies, using a data store management application with a mapping function that translates key values into specific partitions based on customizable approaches, such as modulo operations or hashing functions, and considers data access history to optimize data distribution.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If traditional fixed partitioning methods are used, then data storage structure is simple, but flexibility in managing partitions and optimizing performance is poor

Engineering Contradiction:

Improveflexibility in managing partitionsVSAvoidcomplexity of partitioning system

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent implements dynamic partitioning where the number of partitions, partitioning keys, and mapping methodologies can be changed at runtime without system downtime. The partitioning system adapts to changing data patterns and performance requirements by allowing administrators to modify partition configurations dynamically, resolving the contradiction between flexibility and complexity through controlled adaptability.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system allows changing partitioning parameters such as the number of partitions, partitioning keys, and mapping functions without requiring system restart. This parameter flexibility enables optimization of performance and data distribution while maintaining system operation, addressing the need for adaptability without proportionally increasing operational complexity.

Inventive Principle:

Principle #35Parameter changes

2Ease of operation

If data is partitioned into multiple fixed tables, then data manageability improves, but difficulty in reconfiguring partitions increases

Engineering Contradiction:

Improveease of data managementVSAvoidease of reconfiguration

Core Design Contradiction:

Ease of operationVSEase of repair

Solution Approach 1:

The patent enables dynamic reconfiguration of partition tables where administrators can add, remove, or modify partitions without system downtime. The mapping function automatically adapts to new configurations, making reconfiguration as easy as initial setup and resolving the contradiction between ease of management and ease of reconfiguration.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent introduces a mapping function as an intermediary layer between data and physical partitions. This mapping function handles the complexity of reconfiguration internally, allowing administrators to modify partition structures without directly managing complex data redistribution, thus making reconfiguration as easy as data management.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Productivity

If partitioning keys and mapping methodologies are fixed, then system stability is maintained, but ability to optimize performance decreases

Engineering Contradiction:

Improveperformance optimization capabilityVSAvoidstability of partitioning structure

Core Design Contradiction:

ProductivityVSStability of the object's composition

Solution Approach 1:

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system uses data access history and performance metrics as feedback to guide partitioning optimizations. The mapping function can be adjusted based on observed data patterns, allowing performance optimization while maintaining system stability through measured, data-driven changes rather than arbitrary modifications.

Inventive Principle:

Principle #23Feedback

4Adaptability or versatility

If partition growth is uncontrolled, then adding partitions is simple, but system scalability and performance control deteriorate

Engineering Contradiction:

ImprovescalabilityVSAvoidperformance control

Core Design Contradiction:

Adaptability or versatilityVSEase of operation

Solution Approach 1:

The patent implements controlled partition growth using data access history and performance metrics as feedback mechanisms. The system monitors data patterns and automatically or semi-automatically determines when and how to add partitions, enabling scalability while maintaining performance control through measured growth rather than uncontrolled expansion.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The system performs preliminary analysis of data patterns and access history before adding new partitions. This preliminary action ensures that partition growth is optimized for expected data patterns and performance requirements, maintaining controllability while enabling scalable growth.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS9235613B2Flexible partitioning of data

Publication Date: 2016.01.12 AMAZON TECH INC

AI summary

Disclosed are various embodiments of a flexible application-level partitioning scheme for data storage components. A data store is accessible by one or more computing devices and includes multiple partitions. One of the partitions is identified by mapping a key value of a data item to one of the partitions according to one of multiple versions of mapping metadata. Each of the versions of mapping metadata specifies a respective list of some or all of the partitions and an approach for mapping the key value to one of the partitions. The data item is inserted into the identified partition.