Data Harvester for Compliance Audits

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Large organizations face time-consuming data harvesting processes during compliance audits, which can lead to undetected regulatory breaches and potential financial and legal liabilities due to the lengthy duration of data harvesting, filtering, and analysis across multiple data sources.

Innovation Solution

A data harvester characterizes data sources, samples data to determine the likelihood of successful harvesting, estimates time for completion using machine learning based on previous runs, and provides recommendations for adjustments, allowing for targeted sampling and efficient data collection while generating compliance reports.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If data harvesting is performed across multiple data sources to ensure comprehensive compliance auditing, then the completeness and reliability of compliance detection is improved, but the time required for the audit process increases significantly

Engineering Contradiction:

Improvecompliance detection completenessVSAvoidaudit process duration

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by characterizing data sources and performing initial sampling before the main data harvesting process. This preliminary characterization includes analyzing data source metadata, estimating harvestability, and predicting completion times, which allows for better planning and optimization of the subsequent full harvesting process, ultimately reducing overall audit time while maintaining completeness

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system applies partial action by performing sampling on subsets of data sources rather than harvesting all data immediately. The sampling phase allows the system to identify problematic data sources, estimate completion times, and prioritize harvesting efforts, thereby reducing the effective time required for comprehensive compliance auditing while maintaining detection reliability

Inventive Principle:

Principle #16Partial or excessive action

2Reliability

If comprehensive data harvesting is performed across all data sources to detect all potential compliance breaches, then the reliability of compliance detection is improved, but the time required for harvesting increases to months

Engineering Contradiction:

Improvecompliance breach detectionVSAvoiddata harvesting duration

Core Design Contradiction:

ReliabilityVSDuration of action of moving object

Solution Approach 1:

Before initiating comprehensive data harvesting, the system performs preliminary characterization of data sources including metadata analysis, harvestability assessment, and completion time prediction. This preliminary phase enables the system to identify and prioritize data sources, optimize harvesting parameters, and avoid unnecessary processing, thereby maintaining comprehensive breach detection while significantly reducing the overall harvesting duration

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system implements feedback mechanisms during the harvesting process by continuously monitoring progress, adjusting harvesting rates, and re-evaluating completion time estimates based on actual performance data. This feedback loop allows the system to optimize resource allocation and processing speed in real-time, ensuring comprehensive compliance breach detection is achieved in the minimum necessary time

Inventive Principle:

Principle #23Feedback

3Productivity

If sampling is performed to estimate data harvest success and time, then the efficiency of the auditing process is improved, but the measurement precision of harvest success prediction may be reduced

Engineering Contradiction:

Improveauditing process efficiencyVSAvoidharvest success prediction accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The system performs preliminary sampling and characterization before full harvesting, using the sample data to estimate harvest success and predict completion times. By analyzing metadata, data source structure, and sample content, the system can make accurate predictions about overall harvest outcomes without processing the entire dataset, thereby maintaining measurement precision while significantly improving auditing efficiency

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system changes parameters by adjusting sample sizes, sampling depths, and analysis granularity based on data source characteristics and compliance requirements. This parameter optimization allows the system to achieve sufficient prediction accuracy with smaller, more efficient samples, balancing measurement precision with auditing process efficiency

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS11436243B2Data harvester

Publication Date: 2022.09.06 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

A data harvester enhances compliance audits by characterizing data sources, sampling data in one or more of the data sources to determine likelihood of success of the data harvest, estimating time for the data harvest, making recommendations from the samples based on machine learning relating to previous runs, then sampling additional data while estimated expected completion time. The harvested data may then be analyzed and compared to compliance requirements, and a compliance report may be generated.