Streaming Data Detection Using Type-Mapped Pattern Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for detecting sensitive data in data streams, such as credit card numbers and social security numbers, are computationally expensive due to the need for numerous regular expression matching operations across various formats, which also require extensive buffering and consume significant storage/memory resources.
Innovation Solution
The method involves converting sequences of characters into a type-mapped format, using a streaming-friendly search algorithm like the Aho-Corasick algorithm to detect predefined sequences, and applying validation schemes to extract and confirm sensitive data without extensive buffering or regular expression matching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If regular expression matching operations are performed on data streams to detect sensitive data, then detection accuracy is improved, but computational expense and processing time increase significantly
Solution Approach 1:
The patent applies preliminary action by performing type-mapping on characters before the main detection process. Each character is replaced with a type-mapped character representing its category (digit, letter, special character) in advance, which simplifies subsequent pattern matching operations and reduces computational complexity during real-time processing
Solution Approach 2:
The patent changes parameters by transforming the detection approach from exact string matching to type-based pattern matching. By mapping characters to their types and using convergence rules to represent different formats of sensitive data (e.g., credit card numbers with dashes vs. without), the system achieves detection accuracy while significantly reducing the number of regex operations needed
2Reliability
If extensive buffering is used to process data streams, then detection completeness is improved, but storage and memory resources are consumed
Solution Approach 1:
The patent applies segmentation by processing the data stream in incremental chunks rather than buffering the entire stream. The type-mapping and convergence operations can be performed on individual characters or small batches of characters as they arrive, allowing real-time detection without requiring large storage resources
Solution Approach 2:
The patent introduces type-mapped characters as an intermediary representation between the original data stream and the detection logic. This intermediary layer allows the system to process data incrementally while maintaining detection completeness, as the type-mapped form preserves all necessary information for pattern matching without requiring the original full data stream to be retained
Data Source
AI summary
A method by a network device for detecting data in a data stream. The method includes receiving the data stream, where the data stream includes a sequence of original characters, generating a sequence of type-mapped characters corresponding to the sequence of original characters, converging each of two or more consecutive occurrences of a first character in the sequence of type-mapped characters into a single occurrence of the first character, searching for occurrences of one or more predefined sequences of characters in the sequence of type-mapped characters, and responsive to finding an occurrence of any of the one or more predefined sequences of characters, extracting a sequence of characters in the sequence of original characters corresponding to the occurrence of the predefined sequence of characters found in the sequence of type-mapped characters.