Evolving File Classification for Risk-Aware Data Remediation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data remediation processes face challenges in accurately classifying and managing data, leading to potential disruptions in business operations and legal/regulatory risks due to indiscriminate data deletion.
Innovation Solution
An evolving classification model is employed to classify data, with unclassified files placed in quarantine for later reevaluation, allowing the model to improve over time through retraining, and applying appropriate retention policies once classification confidence is achieved.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is removed to reduce storage costs and improve efficiency, then storage costs and workflow efficiency are improved, but business operations may be impeded and legal/regulatory risks increase
Solution Approach 1:
The system performs preliminary classification of data before removal using an evolving AI model. Data is pre-sorted into categories (retain, archive, delete) based on predicted value and characteristics, allowing organized removal that prevents business disruption while reducing storage costs.
Solution Approach 2:
The classification model continuously learns from feedback loops where classified data is reviewed and feedback is incorporated to improve model accuracy. This ensures more reliable classification over time, reducing misclassification risks to business operations while maintaining cost reduction benefits.
2Productivity
If data is removed to eliminate redundant and obsolete data, then storage costs and workflow efficiency are improved, but legal and regulatory implications arise
Solution Approach 1:
The system applies different classification criteria and retention policies to different data types and locations. Sensitive data receives specialized handling while less critical data can be removed more aggressively, maintaining workflow efficiency while mitigating legal and regulatory risks through differentiated data management.
Solution Approach 2:
Legal and regulatory requirements are incorporated into the preliminary classification process. The model predicts data value and applicability to retention policies before removal, ensuring compliant data management that maintains productivity while avoiding legal issues.
3Measurement precision
If an AI classification model is used to automate data sorting, then classification accuracy and speed are improved, but uncertainty remains about unclassified data
Solution Approach 1:
The classification model is dynamic and evolves over time through continuous learning and retraining. It adapts to new data patterns and organizational needs, improving classification accuracy and confidence in unclassified data through iterative improvement rather than static predictions.
Solution Approach 2:
The system incorporates feedback mechanisms where classification results are reviewed and used to retrain the model. This feedback loop continuously improves classification accuracy and reduces uncertainty about unclassified data by learning from past performance and correcting errors.
Data Source
AI summary
This disclosure describes techniques for performing data remediation. In one example, this disclosure describes a method that includes identifying a plurality of stale files; applying a classification model to each of the plurality of stale files; identifying a plurality of unclassified files, wherein each of the unclassified files is one of the plurality of stale files that the classification model was not able to classify with a confidence level that exceeds a threshold confidence level; updating the classification model, over a period of time, to generate an evolved classification model; applying the evolved classification model to each of the unclassified files; identifying a subset of the unclassified files that the evolved classification model was not able to classify with a confidence level that exceeds the threshold confidence level; and deleting each of the files in the subset of the unclassified files.