Snapshot Data Reconciliation Using Parallel Database Pools

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Reconciling large quantities of enterprise data for analytics and development is time-consuming, resource-intensive, and prone to errors, often requiring manual processes and lacking automation, especially when pre-emptive error detection is necessary.

Innovation Solution

An automated snapshot reconciliation process with multi-threading and logging enhancements, utilizing parallel execution pools, dynamic data masking, and support pipelines to automate repetitive tasks, enabling concurrent processing and execution across multiple database pools.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If manual data reconciliation processes are used, then data accuracy can be maintained through human review, but time consumption and resource intensity increase significantly

Engineering Contradiction:

Improvedata accuracyVSAvoidreconciliation time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system implements automated self-service data reconciliation where the computer system automatically compares baseline data with current data, identifies discrepancies, and generates notifications without requiring manual human intervention for routine reconciliation tasks

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

Manual mechanical review processes are replaced with automated computational systems that use algorithms to compare datasets, detect discrepancies, and generate reports, substituting human labor with automated processing

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Reliability

If pre-emptive error detection is implemented, then data quality issues can be caught early, but computational resources and processing time increase

Engineering Contradiction:

Improveerror detection capabilityVSAvoidcomputational resources

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The system performs preliminary data validation and discrepancy detection by comparing current data against baseline data before errors propagate through the system, catching issues early in the data lifecycle

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system dynamically adjusts processing parameters and thresholds for discrepancy detection based on data characteristics and error patterns, optimizing computational resource usage while maintaining detection effectiveness

Inventive Principle:

Principle #35Parameter changes

3Productivity

If multiple database pools are used for parallel processing, then reconciliation speed increases, but system complexity increases

Engineering Contradiction:

Improvereconciliation speedVSAvoidsystem architecture complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The data reconciliation task is divided into multiple segments processed by separate database pools, with each pool handling specific portions of the data comparison and discrepancy detection independently

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Multiple database pools are implemented with standardized interfaces and protocols, allowing them to perform identical reconciliation functions simultaneously while maintaining system manageability through uniform architecture patterns

Inventive Principle:

Principle #6Universality (Multi-functionality)

4Extent of automation

If automated reconciliation processes are implemented, then manual labor is reduced, but initial setup complexity and resource requirements increase

Engineering Contradiction:

Improveautomation levelVSAvoidsetup complexity

Core Design Contradiction:

Extent of automationVSDevice complexity

Solution Approach 1:

The system creates and maintains baseline data copies that serve as reference standards for automated comparison, enabling automated reconciliation without requiring complex real-time validation logic

Inventive Principle:

Principle #26Copying

Solution Approach 2:

An automated notification system serves as an intermediary between the data reconciliation process and human operators, automatically generating and distributing discrepancy reports without requiring manual intervention in the core processing logic

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS12547604B2Computer system and method for reconciling data

Publication Date: 2026.02.10 THE TORONTO DOMINION BANK

AI summary

A system and method are provided for reconciling data used by a data management system. In some examples, the data management system provides a workspace for machine learning. The method includes obtaining an input dataset, the input dataset being replicated from a baseline dataset to enable the data management system to operate on the input dataset. The method also includes comparing the input dataset to the baseline data set to determine discrepancies between the input and baseline datasets, by, for each of a plurality of database pools, process data assigned to that pool by concurrently checking for the discrepancies and executing statements without waiting for all pools to have finished processing. The method also includes creating a delta table for each pool to identify extracted data associated with the discrepancies; and combining delta pools from the plurality of database pools and process columns in the delta table.