PII Extraction Rule Generation for Machine Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in accurately and efficiently identifying personally identifiable information (PII) in machine-generated data, particularly in unstructured or unknown structured data, which can lead to improper identification or overlooking of PII, especially in large datasets, and require significant computing resources.
Innovation Solution
A method is developed to automatically generate extraction rules using known examples of PII values, allowing the system to identify PII in various contexts and patterns within raw machine data, enabling efficient and accurate detection of PII across multiple occurrences and events.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If machine-generated data is pre-processed and extracted to reduce size, then storage space is saved, but data availability for future use is reduced
Solution Approach 1:
The patent segments data processing into two stages: initial ingestion of raw machine-generated data for comprehensive analysis, and subsequent extraction of specific fields for storage. This allows the system to retain full data availability while reducing storage requirements through selective extraction of only necessary fields.
Solution Approach 2:
The system performs preliminary actions by extracting and storing only specific fields that are likely to contain PII or are needed for analysis, while retaining the ability to access the full raw data when necessary. This preliminary field selection reduces storage burden while maintaining data availability for future queries.
2Reliability
If systems analyze machine-generated data to identify PII, then security and privacy are improved, but computing resources required increase
Solution Approach 1:
The system performs preliminary extraction of specific fields that are likely to contain PII before comprehensive analysis. By pre-identifying and isolating potential PII fields, the system reduces the computing resources needed for subsequent analysis while maintaining high identification accuracy.
Solution Approach 2:
The patent applies local quality by focusing analysis resources on specific fields and data regions that are most likely to contain PII, rather than uniformly processing the entire dataset. This targeted approach improves PII identification accuracy while reducing overall computing resource consumption.
3Reliability
If systems process large datasets to ensure comprehensive PII detection, then detection completeness is improved, but processing time increases
Solution Approach 1:
The system segments the data processing task by first extracting and identifying potential PII fields, then performing comprehensive analysis only on those segmented portions. This segmentation enables comprehensive detection while reducing processing time by avoiding unnecessary analysis of non-PII data.
Solution Approach 2:
The patent applies partial action by processing only the portions of data most likely to contain PII, rather than uniformly processing entire datasets. This partial processing approach maintains detection completeness for PII while significantly reducing processing time compared to full dataset analysis.
Data Source
AI summary
Implementations include receiving a user provided example value of personally identifiable information (PII). Occurrences of the received example value are automatically identified in a dataset of events, wherein each occurrence is identified in a portion of raw machine data of a respective event of the events. For each occurrence of the identified occurrences, an extraction rule is generated, which defines a pattern of the occurrence of the example value and is executable to identify PII values in portions of raw machine data of the events using the pattern. Values of the PII are identified in a set of events using a set of extraction rules comprising the extraction rule of a plurality of the occurrences.