AI Data Remediation for Duplicate and Conflicting Unstructured Content

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Organizations face challenges in managing large volumes of unstructured data with anomalies such as duplicates, knowledge conflicts, noise, and biased data, which affect the accuracy and reliability of AI models, leading to inefficiencies and compliance issues.

Innovation Solution

A data management platform using AI models to identify and remediate unstructured data by generating summaries, detecting duplicates and knowledge conflicts, and identifying anomalies, while ensuring data quality through profiling, threshold modeling, and outlier detection.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If AI models are trained on large volumes of unstructured data, then the AI model performance improves, but data quality deteriorates due to anomalies such as duplicates, knowledge conflicts, noise, and biased data

Engineering Contradiction:

ImproveAI model performanceVSAvoiddata quality

Core Design Contradiction:

ReliabilityVSManufacturing precision

Solution Approach 1:

The system performs preliminary data quality assessment and anomaly detection before AI model training. By proactively identifying duplicates, knowledge conflicts, noise, and biased data through automated profiling and threshold modeling, the system prepares clean training data in advance, preventing quality issues from affecting model performance

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces an intermediary data management platform that sits between raw unstructured data and AI models. This platform acts as a mediator by automatically detecting anomalies, generating data profiles, modeling thresholds for quality assessment, and remediating issues before data reaches the AI training pipeline, thus protecting model performance while handling large volumes of unstructured data

Inventive Principle:

Principle #24Intermediary (Mediator)

2Manufacturing precision

If manual data cleaning and validation processes are used to improve data quality, then data accuracy improves, but operational efficiency deteriorates due to increased time and resource consumption

Engineering Contradiction:

Improvedata accuracyVSAvoidoperational efficiency

Core Design Contradiction:

Manufacturing precisionVSProductivity

Solution Approach 1:

The system implements self-service automated data quality management where the platform independently profiles data, detects anomalies using threshold modeling, and remediates issues without human intervention. This replaces manual cleaning processes with autonomous systems that maintain high data accuracy while operating efficiently at scale

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent substitutes mechanical manual data cleaning operations with automated computational systems. By using AI-driven anomaly detection, automated threshold modeling, and programmatic data remediation, the system replaces time-consuming manual processes with efficient automated mechanisms that maintain or improve data accuracy while increasing operational speed

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Measurement precision

If comprehensive data profiling and anomaly detection are performed to identify all data issues, then data quality assessment improves, but system complexity increases due to multiple processing steps

Engineering Contradiction:

Improvedata quality assessmentVSAvoidsystem complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent creates a universal data management platform that performs multiple functions through integrated components. The same system simultaneously profiles data, detects various anomaly types (duplicates, knowledge conflicts, noise, bias), models thresholds, and remediates issues, reducing overall system complexity by consolidating functions rather than using separate specialized systems for each task

Inventive Principle:

Principle #6Universality (Multi-functionality)

4Stability of the object's composition

If automated remediation processes are implemented to correct data anomalies, then data consistency improves, but resource intensity increases due to computational requirements

Engineering Contradiction:

Improvedata consistencyVSAvoidcomputational resources

Core Design Contradiction:

Stability of the object's compositionVSUse of energy by moving object

Solution Approach 1:

The system applies partial remediation by focusing computational resources on detecting and correcting the most critical anomalies that significantly impact data consistency. Through threshold modeling, the system identifies and remediates high-priority issues (such as duplicates and knowledge conflicts) while using fewer resources on less critical anomalies, achieving improved data consistency with optimized resource consumption

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS20260111395A1Remediation of unstructured data using artificial intelligence

Publication Date: 2026.04.23 CITIBANK N A

AI summary

The systems and methods disclosed herein obtain (e.g., via a user interface) a collection of unstructured data, where each document includes a content set. Using a first AI model set, multiple summaries are generated by categorizing each document into clusters based on vector comparisons of content sets and summarizing the content for each cluster. A second AI model set (same as or different from the first AI model set) identifies duplicate content within the unstructured data by generating similarity values between pairs of summaries and determining if the similarity values meet a predefined threshold. A report is generated (e.g., on the user interface) indicating the duplicate content sets and/or the collection of unstructured data.