Data Quality Analysis System for Source Data Cleansing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data migration processes face challenges in quantifying data quality across multiple source systems, leading to poor data quality and costly delays due to the complexity of determining data quality metrics and missing data requirements for target systems, which results in inefficient data cleansing and harmonization.
Innovation Solution
A system that determines domain scores for source data based on quality metrics for a target system, identifies corresponding processes, and cleanses data accordingly, ensuring data quality meets target system requirements through a processor-driven approach that includes a source analysis phase, target process phase, and load analysis phase.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data quality metrics are determined across multiple source systems, then data quality assessment becomes comprehensive, but the complexity of the process increases significantly
Solution Approach 1:
The patent segments the complex multi-source data quality assessment into distinct phases: source analysis phase (assessing data in source systems), target process phase (evaluating data requirements for target processes), and load analysis phase (validating data for loading). Each phase focuses on specific assessment tasks, making the overall complex process more manageable and systematic.
Solution Approach 2:
The patent introduces an intermediary data quality assessment framework that bridges source systems and target systems. This framework includes data quality rules, metrics definitions, and assessment protocols that mediate between diverse source data formats and target system requirements, reducing the direct complexity of cross-system assessment.
2Reliability
If data cleansing is performed earlier in the transformation process, then data quality improves, but the time and resources required increase
Solution Approach 1:
The patent performs data quality assessment and identifies cleansing requirements in advance during the source analysis and target process phases, before actual data transformation and loading. This preliminary identification of data quality issues allows for proactive cleansing planning and execution, improving data quality without causing unexpected delays during the critical loading phase.
Solution Approach 2:
The patent implements feedback mechanisms where data quality assessment results from the source analysis and target process phases inform the data cleansing activities. This feedback loop allows for targeted cleansing based on actual assessed quality metrics rather than generic preprocessing, optimizing the balance between quality improvement and time investment.
3Productivity
If data quality metrics are not determined early, then project timelines are maintained, but data quality suffers leading to business process interruptions
Solution Approach 1:
The patent segments the project timeline into distinct phases with embedded data quality assessment activities. The source analysis phase and target process phase include quality metric determination as integral components, allowing quality assessment to occur systematically throughout the project rather than as a separate time-consuming add-on, thus maintaining timeline while improving quality.
Solution Approach 2:
The patent creates a universal data quality assessment framework that serves multiple functions: it assesses source data quality, evaluates target system requirements, identifies cleansing needs, and validates loaded data. This multi-functional approach consolidates what could be separate time-consuming activities into an integrated process that maintains project timelines while comprehensively addressing data quality.
Data Source
AI summary
A system transfers data between source systems and a target system. The system determines a domain score for data domains of source data from the source systems based on data quality metrics for the target system. The domain score indicates data quality with respect to the target system. Corresponding processes of the target system are identified for the data domains, and a process score is determined for the identified processes based on a corresponding domain score. The process score indicates data quality with respect to the identified processes. The system cleanses the source data based on the domain score and/or process score, and validates the cleansed source data against the target system for transference. Embodiments of the present invention further include a method and computer program product for transferring data between source systems and a target system in substantially the same manner described above.