AI Remediation of Unstructured Data for Duplicate and Conflict Detection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Organizations face challenges in managing large volumes of unstructured data, including duplicates, knowledge conflicts, noise, and biased data, which affect the accuracy and reliability of AI models, leading to inefficiencies and compliance issues.

Innovation Solution

A data management platform using AI models to identify and remediate unstructured data by generating summaries, detecting duplicates and knowledge conflicts, and identifying anomalies, while also determining out-of-distribution data to improve data quality and reduce biases.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If AI models are trained on large volumes of unstructured data, then the model's predictive capability is improved, but data quality deteriorates due to duplicates, conflicts, noise, and biases

Engineering Contradiction:

ImproveAI model accuracyVSAvoiddata quality

Core Design Contradiction:

ReliabilityVSManufacturing precision

Solution Approach 1:

The patent applies preliminary action by performing data remediation before AI model training. The system proactively identifies and resolves data quality issues (duplicates, conflicts, noise, biases) in advance, ensuring that only high-quality data is used for training. This prevents poor quality data from degrading model accuracy while still enabling comprehensive data utilization.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent implements self-service through automated data quality assessment and remediation. The system uses AI models to automatically detect duplicates, identify knowledge conflicts, filter noise, and correct biases in unstructured data without requiring manual intervention. This self-service approach maintains data quality while processing large volumes of data efficiently.

Inventive Principle:

Principle #25Self-service

2Manufacturing precision

If manual data cleaning processes are used to remove duplicates and conflicts, then data quality is improved, but processing time increases

Engineering Contradiction:

Improvedata qualityVSAvoidprocessing time

Core Design Contradiction:

Manufacturing precisionVSLoss of time

Solution Approach 1:

The patent replaces manual mechanical data cleaning processes with automated AI-based systems. Instead of human operators manually reviewing and cleaning data, the system uses machine learning models to automatically detect duplicates, resolve conflicts, and filter noise. This substitution dramatically reduces processing time while maintaining or improving data quality.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent applies parameter changes by transforming unstructured data into structured representations that can be efficiently processed. The system converts various unstructured data formats into standardized structures with defined parameters, enabling automated comparison, matching, and quality assessment. This transformation accelerates the cleaning process while ensuring comprehensive quality control.

Inventive Principle:

Principle #35Parameter changes

3Loss of information

If all unstructured data is processed and analyzed, then completeness is improved, but computational resources are overwhelmed

Engineering Contradiction:

Improvedata completenessVSAvoidcomputational resources

Core Design Contradiction:

Loss of informationVSUse of energy by moving object

Solution Approach 1:

The patent applies segmentation by dividing the large volume of unstructured data into smaller, manageable segments or batches. The system processes data in distributed chunks across multiple computational nodes, analyzing each segment independently before aggregating results. This approach maintains complete data coverage while distributing computational load to avoid overwhelming resources.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent implements partial action by prioritizing the processing of high-impact data segments first. The system identifies and processes data portions that have the greatest influence on AI model training and data quality, while deferring or using simplified methods for lower-priority segments. This selective approach ensures essential completeness while conserving computational resources.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS12511264B2Remediation of unstructured data using artificial intelligence

Publication Date: 2025.12.30 CITIBANK N A

AI summary

The systems and methods disclosed herein obtain (e.g., via a user interface) a collection of unstructured data, where each document includes a content set. Using a first AI model set, multiple summaries are generated by categorizing each document into clusters based on vector comparisons of content sets and summarizing the content for each cluster. A second AI model set (same as or different from the first AI model set) identifies duplicate content within the unstructured data by generating similarity values between pairs of summaries and determining if the similarity values meet a predefined threshold. A report is generated (e.g., on the user interface) indicating the duplicate content sets and/or the collection of unstructured data.