Direct Data Distribution in Analysis Systems

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Existing data synchronization methods in data analysis systems are inefficient as they often require proxying data through a single target node, leading to increased data transfer and bandwidth usage, especially when dealing with large datasets.

Innovation Solution

A method that involves determining a distribution key for data records to directly connect and send change records to specific target database nodes within a data analysis system, eliminating the need for proxying through a single node and optimizing bandwidth usage by establishing direct connections between the source and target database nodes.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If data is proxied through a single target node for synchronization, then data can be transferred to multiple locations, but data transfer bandwidth increases and efficiency decreases

Engineering Contradiction:
Improvedata synchronization efficiencyVSAvoidbandwidth usage
Core Design Contradiction:
ProductivityVSLoss of energy

Solution Approach 1:

The patent segments the data distribution process by introducing multiple target database nodes that independently receive and process change records. Instead of funneling all data through a single proxy node, the system divides the replication workload across multiple parallel paths, each with its own dedicated connection. This segmentation eliminates the bottleneck at the single proxy node and allows simultaneous data transfer to multiple target nodes, directly resolving the bandwidth congestion issue.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The patent introduces distribution keys as intermediary elements that mediate between the source database system and multiple target database nodes. The distribution key determines which target node receives which change record, enabling intelligent routing that avoids unnecessary data transfer. This intermediary mechanism allows the system to precisely direct data to appropriate targets without requiring a single proxy node to handle all traffic, thereby reducing overall bandwidth consumption.

Inventive Principle:
Principle #24Intermediary (Mediator)

2Device complexity

If a single target node proxies all data, then system architecture is simplified, but data transfer overhead increases

Engineering Contradiction:
Improvesystem architectureVSAvoiddata transfer time
Core Design Contradiction:
Device complexityVSLoss of time

Solution Approach 1:

The patent segments the replication architecture into multiple independent source-target connections, each with its own dedicated pathway. This segmentation eliminates the sequential bottleneck inherent in single-proxy architectures, allowing parallel data transfer operations. While the architecture becomes slightly more complex with multiple nodes, the elimination of sequential transfer overhead results in significant time savings for large datasets.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The patent transitions from a one-dimensional sequential transfer model (single proxy node handling all data sequentially) to a multi-dimensional parallel transfer model (multiple simultaneous connections). By adding the dimension of parallelism through multiple target nodes, the system achieves substantially faster data synchronization without requiring excessive architectural complexity.

Inventive Principle:
Principle #17Another dimension (Dimensionality change)

3Loss of energy

If direct connections to multiple target nodes are established, then bandwidth usage is optimized, but connection management complexity increases

Engineering Contradiction:
Improvebandwidth efficiencyVSAvoidconnection management
Core Design Contradiction:
Loss of energyVSDevice complexity

Solution Approach 1:

The patent employs distribution keys as intermediary routing mechanisms that automatically determine the appropriate target node for each change record. This intermediary approach eliminates the need for complex manual connection management, as the distribution key-based routing automatically directs data to the correct node. The system manages multiple connections through this automated intermediary mechanism, reducing operational complexity while maintaining high bandwidth efficiency.

Inventive Principle:
Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent implements self-service connection management where the source database system automatically establishes and maintains direct connections to multiple target nodes based on distribution key values. The system autonomously handles connection setup, data routing, and synchronization without requiring external intervention or complex centralized management. This self-service approach optimizes bandwidth usage while keeping connection management complexity at acceptable levels through automation.

Inventive Principle:
Principle #25Self-service

Data Source

PatentUS12259905B2Data distribution in data analysis systems
Publication Date: 2025.03.25 INTERNATIONAL BUSINESS MACHINE CORPORATION
  • US12259905B2 patent drawing
  • US12259905B2 patent drawing
  • US12259905B2 patent drawing

AI summary

The present disclosure relates to a computer implemented method for data synchronization in a data analysis system. The data analysis system comprises a source and target database system. The method comprises: receiving a change record describing an operation performed on a data record in the source database system. The change record may be read for determining a value of a distribution key of the data record. The value of the distribution key may be used for selecting a target database node of the target database system where the operation is to be performed. A direct connection may be established to the selected target database node and the change record may be provided to the selected target database node through the direct connection.