Automated Sensitive Data Detection Using Fuzzy Matching and Culling Rules
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In live service environments, such as modern video games, identifying sensitive data like passwords or credentials within large, heterogeneous datasets is challenging due to their variability in type and format, making manual auditing time-consuming and conventional automated methods ineffective.
Innovation Solution
A computer-implemented method using attention rules to identify data snippets, followed by fuzzy matching and culling rules to categorize and remediate sensitive data, allowing for efficient detection and removal of false positives from true positive buckets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual auditing is used to identify sensitive data, then detection accuracy can be maintained, but time consumption increases significantly
Solution Approach 1:
The patent segments the dataset into multiple partitions and processes them in parallel using multiple processing threads. Each thread independently audits a portion of the data, maintaining detection accuracy while distributing the time burden across multiple concurrent operations, thereby reducing total auditing time.
2Productivity
If conventional automated methods are used to detect sensitive data, then processing speed improves, but detection accuracy deteriorates due to false positives
Solution Approach 1:
The patent implements feedback mechanisms where detection results are continuously refined. Processing threads use feedback from previously identified sensitive data patterns to improve their detection algorithms, reducing false positives while maintaining high processing speed through automated iterative improvement.
Solution Approach 2:
The patent replaces manual mechanical auditing with automated electronic processing systems that use algorithms and heuristics to identify sensitive data. This substitution enables rapid processing of large datasets while incorporating intelligent detection rules that reduce false positives compared to simple keyword matching.
3Productivity
If automated detection is applied to heterogeneous data of variable types and formats, then processing efficiency improves, but the ability to handle diverse data formats deteriorates
Solution Approach 1:
The patent implements universal processing threads capable of handling multiple data types and formats simultaneously. Each processing thread is designed with multi-functional detection capabilities that can adapt to various data structures, file formats, and encoding types, enabling efficient processing of heterogeneous datasets without sacrificing format versatility.
Data Source
AI summary
A computer-implemented method is provided of finding one or more data items that match one or more defined criteria in a dataset. The method comprises identifying data snippets of the dataset using a set of one or more attention rules; categorizing the identified data snippets using fuzzy matching by assigning them to buckets such that each bucket contains data snippets that are similar to another according to a similarity measure; classifying buckets containing data snippets having more than a threshold number of the true positive data items as true positive buckets and remaining buckets as false positive buckets; calculating culling rules based on the true positive buckets and the false positive buckets, and using the culling rules to remove the false positive data items from the true positive buckets.