Sensitive Data Tracking in Big Data via Pattern Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In big data environments, tracking sensitive data is challenging due to its widespread distribution across various storage types and the difficulty in monitoring changes in raw data, which can result in personally identifiable information appearing unexpectedly, hindering effective tracking and access permission management.
Innovation Solution
A system that scans a big data management system to identify sensitive data by matching data patterns with a pattern database, retrieves access permissions, and flags discrepancies, allowing for automated scheduling and background processing to ensure proper permission settings and data security.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If sensitive data is copied and stored in various locations across different types of data storage for various use cases, then data accessibility and versatility are improved, but tracking the sensitive data becomes difficult
Solution Approach 1:
The patent segments the tracking function into distributed tracking objects that are placed at each location where sensitive data is stored. Each tracking object independently monitors its local data, allowing the system to track sensitive data across multiple locations without centralizing the tracking mechanism, thus maintaining accessibility while enabling effective tracking.
Solution Approach 2:
The patent implements feedback mechanisms where tracking objects continuously monitor data locations and report back to a central coordination system. This feedback loop enables the system to maintain awareness of sensitive data locations and access patterns, resolving the tracking difficulty while preserving data accessibility across distributed storage.
2Productivity
If raw data is mapped or derived into output data, then data processing and utility are improved, but sensitive data may appear unexpectedly in output columns or files
Solution Approach 1:
The patent applies preliminary action by placing tracking objects on raw data sources before data processing occurs. These tracking objects establish baseline knowledge of sensitive data locations and characteristics, enabling the system to proactively identify and control sensitive data as it transforms through processing pipelines, preventing unexpected appearance in output while maintaining processing productivity.
Solution Approach 2:
The tracking system implements continuous feedback during data processing transformations. As raw data is mapped or derived into output data, the tracking objects monitor changes and report sensitive data appearances in real-time, allowing the system to maintain control over sensitive data throughout the processing pipeline rather than only at static points.
3Measurement precision
If automated scanning and pattern matching is implemented to detect sensitive data, then detection capability is improved, but system complexity increases
Solution Approach 1:
The patent segments the complex detection system into simple, standardized tracking objects that are deployed at data locations. Each tracking object uses basic pattern matching against a pattern database, breaking down the complex detection task into many simple, manageable units that collectively achieve high detection precision without requiring a monolithic complex system.
Solution Approach 2:
The patent uses copying by replicating standardized tracking object templates across multiple data locations. Rather than designing unique complex detection logic for each location, the system copies proven tracking object designs, simplifying system implementation while maintaining consistent detection capability across the entire data environment.
Data Source
AI summary
A system may retrieve a pattern from a pattern database with the pattern identifying a type of sensitive data. The system may also retrieve data identified by a variable from a big data management system. The system may then match the data to the pattern to detect the type of sensitive data in the data. An output may be generated in response to the data matching the pattern. A variable access permission may be retrieved for the variable from a permissions repository, a sensitive data permission may be retrieved for the type of sensitive data from the permissions repository, and the variable access permission may be compared to the sensitive data permission to detect a discrepancy.