Data Classification via Flow Confidence Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data classification methods struggle to efficiently classify data that has undergone processing operations, leading to challenges in compliance with regulatory requirements and increased costs.
Innovation Solution
A system utilizing data flow analysis to perform data classification, which involves receiving a source dataset, generating a derived dataset through processing operations, calculating flow confidence using an adaptive algorithm, and determining tag confidence for each tag in the derived dataset, with actions taken based on confidence thresholds and human review when necessary.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional data classification methods are used on processed data, then classification can be performed, but time and cost increase significantly
Solution Approach 1:
The system performs preliminary classification on source data before processing operations occur. By pre-classifying the source dataset and storing these classifications, the system avoids the need to re-classify data after transformations, significantly reducing classification time and computational cost while maintaining accuracy.
Solution Approach 2:
The system segments the classification process into distinct phases: source data classification, processing operation analysis, and derived data classification inference. This segmentation allows each phase to be optimized independently, with the source classification being performed once and reused across multiple derived datasets.
2Reliability
If traditional data classification methods are used on processed data, then classification can be performed, but costs increase
Solution Approach 1:
The system implements feedback mechanisms where classification results from source data are fed forward to inform the classification of derived data. The processing operations are analyzed and their impact on data characteristics is fed back into the classification logic, allowing the system to maintain high compliance reliability without requiring complete re-classification.
Solution Approach 2:
By performing classification actions preliminarily on source data and storing these results, the system eliminates the need for complex re-classification operations on derived data. This preliminary classification approach simplifies the overall system complexity while maintaining compliance reliability through systematic reuse of classification results.
3Reliability
If classification is performed on all derived data, then compliance is ensured, but processing time increases
Solution Approach 1:
The system performs partial classification actions by classifying only the source data fully and then inferring classifications for derived data based on processing operation analysis. This partial approach to re-classification maintains regulatory compliance for all data while avoiding the excessive processing time that would result from fully re-classifying every derived dataset.
Solution Approach 2:
By pre-classifying source data and storing these results for later reuse, the system ensures regulatory compliance is established upfront. This preliminary compliance action allows subsequent derived datasets to leverage these pre-established classifications, maintaining compliance reliability while preserving data processing throughput.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Described herein is a system and method for utilizing data flow analysis to perform data classification with respect to a source dataset and a generated derived dataset. A flow confidence for a field is calculated using an adaptive algorithm in accordance with the action performed and the derived dataset. An associated derived confidence for a particular tag is calculated in accordance with an associated confidence and the flow confidence. When the associated derived confidence is greater than or equal to a first threshold, the particular tag is copied to the derived dataset. In some embodiments, when the associated derived confidence is less than or equal to a second threshold, the particular tag is not copied to the derived dataset. Otherwise an action to be taken is identified. A response to the action is received and the adaptive algorithm is modified in accordance with the received response.