Data Partitioning System for Live Dataset Transfer

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Transferring large data sets between computing systems is challenging due to resource limitations, making manual partitioning difficult, especially for live or actively modified datasets, which can result in uneven system performance, backups, slowdowns, and crashes.

Innovation Solution

A data partitioning and transferring system (DPS) automatically determines optimal partition sizes and numbers based on available resources and dataset characteristics, using unique columns and profiling results to ensure equal or roughly equal partition sizes, thereby maximizing resource utilization and preventing system interruptions.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If manual partitioning is attempted for large datasets, then data can be divided into smaller units, but the process becomes difficult or impossible to accomplish successfully

Engineering Contradiction:

Improvedata sizeVSAvoidpartitioning process

Core Design Contradiction:

Quantity of substanceVSEase of operation

Solution Approach 1:

The system enables self-service automatic partitioning where the database management system autonomously analyzes dataset characteristics, determines optimal partitioning strategies, and executes the partitioning without administrator intervention. This resolves the contradiction by making the system self-sufficient for handling large data partitioning tasks.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent replaces manual mechanical partitioning operations with automated computational algorithms that analyze data patterns, calculate optimal partition sizes, and execute divisions. This substitution eliminates the difficulty of manual partitioning while effectively managing large datasets.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Ease of operation

If data is transferred as a single block, then transfer simplicity is maintained, but resource limitations prevent successful transfer

Engineering Contradiction:

Improvetransfer processVSAvoiddata size

Core Design Contradiction:

Ease of operationVSQuantity of substance

Solution Approach 1:

The system automatically segments large datasets into smaller partitions based on resource constraints and data characteristics. This segmentation enables transfer operations to proceed successfully by breaking down the overwhelming single-block transfer into manageable units while maintaining operational simplicity through automation.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The partitioning strategy is dynamically adjusted based on available resources, data patterns, and transfer requirements. The system adapts partition sizes and numbers automatically, providing both simplicity and resource efficiency without requiring manual configuration.

Inventive Principle:

Principle #15Dynamics

3Quantity of substance

If manual partitioning is performed on live datasets, then data can be divided, but system performance degrades with backups and slowdowns

Engineering Contradiction:

Improvedata partitioningVSAvoidsystem performance

Core Design Contradiction:

Quantity of substanceVSReliability

Solution Approach 1:

The system performs preliminary analysis of live datasets to determine optimal partitioning strategies before execution. It identifies data patterns, hotspots, and access patterns in advance, allowing partitioning to be performed with minimal disruption to ongoing operations and maintaining system reliability.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system incorporates feedback mechanisms that monitor system performance during partitioning operations on live datasets. It adjusts partitioning strategies in real-time based on observed performance metrics, preventing degradation and ensuring reliable operation throughout the partitioning process.

Inventive Principle:

Principle #23Feedback

4Ease of operation

If uneven partitions are created, then manual partitioning is simpler, but system performance becomes inconsistent

Engineering Contradiction:

Improvepartition creationVSAvoidpartition size uniformity

Core Design Contradiction:

Ease of operationVSStability of the object's composition

Solution Approach 1:

The system automatically adjusts partition parameters including size, number, and distribution based on data characteristics and system resources. This dynamic parameter optimization ensures uniform partition sizes that provide consistent system performance while eliminating the need for manual intervention.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS11151128B2Data partitioning and transfer system

Publication Date: 2021.10.19 SAP SE

AI summary

Disclosed herein are system, method, and computer program product embodiments for providing data partitioning and transferring operations. An embodiment operates by determining a partition size and a number of partitions for an initial data set to be transferred from a first location to a second location. A uniqueness factor for at least a subset of the columns of the dataset is determined, and a set of unique columns is identified from the initial data set based on the uniqueness factor. Based on the partition size, a set of values from the row records from the set of unique columns is identified. Based on the identified set of values, the initial data set is partitioned into the number of partitions. One of transmitting or receiving at least one of the partitions is performed.