Proactive Duplicate Identification in ETL Pipelines

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data warehouse systems face inefficiencies in handling and processing large volumes of data due to the presence of duplicate information, which leads to increased storage costs, resource utilization, and processing time, particularly in operations like queries and reporting.

Innovation Solution

A proactive duplicate identification system and method that employs a hardware-implemented document data extractor, indexer, analyzer, validator, and report generator to dynamically identify and eliminate duplicates in real-time during the ETL process, utilizing core and fuzzy matching rules to determine duplicate documents and prevent fraudulent data entry.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If data is loaded from multiple sources into the database without proactive duplicate identification, then data volume and storage capacity increase, but processing time and resource utilization increase due to duplicate information

Engineering Contradiction:
Improvedata volumeVSAvoidprocessing time
Core Design Contradiction:
Quantity of substanceVSProductivity

Solution Approach 1:

The system performs duplicate identification during the ETL transformation phase, before data is loaded into the database. The document data extractor, indexer, and analyzer proactively detect duplicates by comparing incoming documents against existing database records and each other, eliminating duplicates before they consume database storage and processing resources.

Inventive Principle:
Principle #10Preliminary action

Solution Approach 2:

The system extracts and removes duplicate documents from the data stream during the transformation phase. By identifying and eliminating duplicates before database insertion, the system prevents duplicate data from entering the database, thereby reducing storage requirements and improving query performance without sacrificing data completeness.

Inventive Principle:
Principle #2Taking out (Extraction)

2Reliability

If duplicate identification is performed after data entry, then data is already stored in the database, but storage and processing inefficiencies remain

Engineering Contradiction:
Improvedata accuracyVSAvoidresource utilization
Core Design Contradiction:
ReliabilityVSLoss of energy

Solution Approach 1:

The system performs duplicate identification during the ETL transformation phase, before data is loaded into the database. The document data extractor, indexer, and analyzer proactively detect duplicates by comparing incoming documents against existing database records and each other, eliminating duplicates before they consume database storage and processing resources.

Inventive Principle:
Principle #10Preliminary action

3Productivity

If proactive duplicate identification is implemented during ETL, then storage and processing inefficiencies are reduced, but system complexity increases

Engineering Contradiction:
Improvedata processing efficiencyVSAvoidsystem complexity
Core Design Contradiction:
ProductivityVSDevice complexity

Solution Approach 1:

The duplicate identification system is divided into distinct modular components: a document data extractor that retrieves data, an indexer that organizes it, and an analyzer that compares records. Each component performs a specific function in the duplicate detection process, making the overall system more manageable and maintainable while improving processing efficiency.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The system introduces an intermediary duplicate identification layer between the data source and the database. This layer includes the extractor, indexer, and analyzer components that act as mediators to filter and validate data before it enters the database, reducing the burden on the database system itself.

Inventive Principle:
Principle #24Intermediary (Mediator)

4Quantity of substance

If large volumes of data are stored without duplicate elimination, then data completeness is maintained, but storage costs and resource utilization increase

Engineering Contradiction:
Improvedata storage capacityVSAvoidstorage costs
Core Design Contradiction:
Quantity of substanceVSLoss of energy

Solution Approach 1:

The system extracts and removes duplicate documents from the data stream during the transformation phase. By identifying and eliminating duplicates before database insertion, the system prevents duplicate data from entering the database, thereby reducing storage requirements and improving query performance without sacrificing data completeness.

Inventive Principle:
Principle #2Taking out (Extraction)

Data Source

PatentUS10282353B2Proactive duplicate identification
Publication Date: 2019.05.07 ACCENTURE GLOBAL SERVICES LTD
  • US10282353B2 patent drawing
  • US10282353B2 patent drawing
  • US10282353B2 patent drawing

AI summary

Proactive duplicate identification may include receiving a document, and determining data related to the document. The documents may be indexed from history data and/or current data for duplicate analysis of the document. The indexing of the documents may include an initial indexing based on the determination of the data related to the document for an initial data field, and a further indexing based on a further determination of further data related to the document for a further data field. The initial indexing may narrow a scope of the documents for the duplicate analysis, and the further indexing may further narrow the scope of the documents for the duplicate analysis. The indexed documents may be used to perform the duplicate analysis of the document to determine whether the document is a duplicate document, a probable duplicate document, or a non-duplicate document.