Parallel Database Hash Sum Comparison Across Disparate Systems

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing methods for comparing databases with different parallel system architectures face challenges in ensuring byte-for-byte equivalence, as they often require data to be transferred to a single node, leading to bottlenecks and inefficiencies, especially when dealing with large and disparate systems.

Innovation Solution

The method calculates a hash sum for each node in a database with a parallel system architecture, combining these node hash sums to generate a database hash sum, allowing for efficient comparison across different architectures, including the use of multiple aggregation functions to account for duplicate records.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If data is transferred to a single node for comparison, then comparison accuracy is improved, but processing time increases and system bottleneck occurs

Engineering Contradiction:

Improvecomparison accuracyVSAvoidprocessing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The database comparison task is segmented and distributed across multiple nodes. Each node calculates a local hash sum for its portion of the data, and these local hash sums are then combined to produce the overall database hash sum. This segmentation allows parallel processing while maintaining comparison accuracy.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The approach transitions from a single-node sequential processing model to a multi-node parallel processing model. By adding the dimension of distributed computation across multiple nodes, the system achieves both high accuracy and reduced processing time through parallel hash sum calculations.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Measurement precision

If data is transferred to a single node for comparison, then comparison accuracy is improved, but device complexity increases

Engineering Contradiction:

Improvecomparison accuracyVSAvoidsystem complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The hash sum calculation mechanism is designed to be universally applicable across different parallel system architectures. The same basic principle of calculating node hash sums and combining them works for various database systems, reducing the need for architecture-specific complex handling while maintaining accuracy.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The node hash sum acts as an intermediary that simplifies the comparison process. Instead of requiring complex direct comparison between disparate parallel systems, the node hash sum serves as a standardized intermediate representation that can be combined across different architectures to produce a comparable result.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Ease of operation

If traditional comparison techniques are used, then ease of operation is maintained, but reliability decreases for disparate parallel systems

Engineering Contradiction:

Improveoperation simplicityVSAvoidcomparison reliability

Core Design Contradiction:

Ease of operationVSReliability

Solution Approach 1:

The approach changes the parameter used for comparison from traditional methods (such as record counts, byte counts, or column sums) to hash sums. This parameter change enables reliable comparison across disparate parallel systems while maintaining ease of operation, as hash sums provide a standardized and robust comparison metric.

Inventive Principle:

Principle #35Parameter changes

4Productivity

If parallel processing is used for hash sum calculation, then productivity is improved, but measurement precision may be affected by duplicate records

Engineering Contradiction:

Improveprocessing speedVSAvoidhash sum accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The node hash sums calculated by individual nodes are merged or combined to produce the overall database hash sum. This merging process ensures that duplicate records across different nodes are properly accounted for, maintaining measurement precision while benefiting from the speed of parallel processing.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The approach uses composite aggregation functions that can handle duplicate records appropriately. By combining multiple aggregation functions with different behaviors, the system can accurately compute hash sums even when duplicate records are present, maintaining both speed and precision.

Inventive Principle:

Principle #40Composite materials

Data Source

PatentUS11182366B2Comparing data stores using hash sums on disparate parallel systems

Publication Date: 2021.11.23 BANK OF AMERICA CORP

AI summary

Aspects described herein relate to methods and systems for comparing data stored in disparate parallel systems using hash sums. A database having a parallel system architecture may comprise a plurality of nodes each storing a plurality of records. A central node may initiate parallel calculation of a set of node hash sums for each individual node. Calculating a node hash sum for an individual node may comprise calculating, by the individual node, a set of hash values for each individual record of the plurality of records stored by the individual node and combining each hash value of the set of hash values to generate the node hash sum for the individual node. The central processor may combine each node hash sum to generate a database hash sum. The central processor may store the generated database hash sum and/or utilize it in comparisons with database hash sums for other databases.