Streaming Data Detection Using Type-Mapped Pattern Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for detecting sensitive data in data streams, such as credit card numbers and social security numbers, are computationally expensive due to the need for numerous regular expression matching operations across various formats, which requires extensive buffering and high computational resources.
Innovation Solution
The method involves converting sequences of characters in a data stream into a type-mapped format, using the Aho-Corasick algorithm to search for predefined sequences, and applying validation schemes to efficiently detect sensitive data without extensive buffering or regular expression matching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If regular expression matching operations are performed on data streams to detect sensitive data, then detection accuracy is improved, but computational cost increases
Solution Approach 1:
The patent performs preliminary actions by converting diverse data formats into a unified type-mapped format before detection, and by pre-processing the data stream to identify potential sensitive data regions. This preliminary format standardization reduces the complexity of subsequent matching operations, allowing accurate detection with lower computational cost during real-time processing.
2Adaptability or versatility
If multiple regular expression matching operations are performed on various data formats, then comprehensive detection coverage is improved, but processing time increases
Solution Approach 1:
The patent changes the parameter of data representation by converting various data formats into a unified type-mapped format. This parameter transformation allows a single detection pass to cover multiple original formats, thereby maintaining comprehensive detection coverage while significantly reducing processing time compared to performing separate matching operations for each format.
3Reliability
If data is buffered for batch processing, then processing completeness is improved, but storage requirements increase
Solution Approach 1:
The patent extracts and processes only the essential characteristics of the data stream in real-time, taking out the critical detection logic from the bulk data processing. By using streaming-friendly techniques that process data incrementally without requiring full buffering, the system achieves processing completeness while minimizing storage requirements to only what is necessary for the detection state.
Data Source
AI summary
A method by a network device for detecting data in a data stream. The method includes receiving the data stream, where the data stream includes a sequence of original characters, generating a sequence of type-mapped characters corresponding to the sequence of original characters, converging each of two or more consecutive occurrences of a first character in the sequence of type-mapped characters into a single occurrence of the first character, inserting beginning/ending of segment indicators in the sequence of type-mapped characters, searching for occurrences of one or more predefined sequences of characters in the sequence of type-mapped characters, and responsive to finding an occurrence of any of the one or more predefined sequences of characters, extracting a sequence of characters in the sequence of original characters corresponding to the predefined sequence of characters found in the sequence of type-mapped characters.