STR&M Table Parallel Matching for Dynamic Signatures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Processing large data sets in the 'Big Data' realm is challenging due to the complexity and size, requiring efficient mechanisms for rapid searching and matching of dynamic signatures, especially in fields like fraud detection and DNA analysis, where traditional methods are inadequate for high-speed processing and frequent signature updates.
Innovation Solution
The implementation of a state transition, result, and mask (STR&M) table mechanism that enables parallel processing across hardware devices, utilizing Regular Expressions and hardware acceleration through mechanisms like OpenCL and CUDA, allowing for rapid matching of input strings against a dynamic set of signatures with minimal disruption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional data processing methods are used to process large data sets, then processing can be performed with simple hardware, but processing speed and efficiency are insufficient for high-speed matching requirements
Solution Approach 1:
The patent divides the large data processing task into multiple parallel matching operations, where each processing element handles a portion of the signatures simultaneously. This segmentation enables high-speed processing by distributing the workload across multiple hardware units, resolving the contradiction between processing speed and hardware complexity.
Solution Approach 2:
The patent transitions from sequential processing to parallel processing by adding a spatial dimension to the computation. Multiple processing elements operate concurrently on different portions of the data, effectively moving from a one-dimensional sequential approach to a multi-dimensional parallel architecture, thereby achieving high speed without excessive complexity.
2Measurement precision
If a dynamic set of signatures is maintained to improve matching accuracy, then detection precision is improved, but the complexity of managing and updating signatures increases
Solution Approach 1:
The patent pre-compiles signatures into efficient data structures (such as Aho-Corasick automata or trie structures) before the matching process begins. This preliminary preparation organizes the dynamic signature set into a form that enables fast simultaneous matching, reducing the complexity of managing dynamic signatures while maintaining high matching accuracy.
Solution Approach 2:
The patent creates optimized copies of the signature set in memory-efficient formats suitable for parallel processing. These copies are loaded into the processing elements, allowing the system to maintain a dynamic signature set without the complexity of managing the original representations, thereby achieving both accuracy and manageable complexity.
3Productivity
If parallel processing is implemented across multiple hardware devices to increase processing speed, then productivity is improved, but system complexity and coordination overhead increase
Solution Approach 1:
The patent merges the coordination function into a single master controller that manages multiple processing elements. This centralization of control simplifies the system architecture by eliminating the need for complex peer-to-peer coordination between processing elements, thereby enabling parallel processing while keeping system coordination complexity manageable.
Solution Approach 2:
The patent designs processing elements with universal functionality that can handle any signature type through a standardized interface. This multi-functionality allows different processing elements to work together seamlessly without requiring complex specialized coordination, simplifying the overall system architecture while maintaining high parallel processing throughput.
4Adaptability or versatility
If frequent signature updates are performed to adapt to changing data patterns, then adaptability is improved, but processing disruptions and time loss increase
Solution Approach 1:
The patent pre-organizes signatures into efficient data structures that can be quickly reloaded or updated without requiring complete system reconfiguration. This preliminary structuring allows frequent signature updates to be performed with minimal processing disruption, maintaining adaptability while reducing time loss.
Solution Approach 2:
The patent enables signature updates to occur without completely stopping the matching process. By maintaining continuous operation and allowing incremental updates to the signature data structures, the system achieves frequent adaptability while minimizing processing disruptions and time loss.
Data Source
AI summary
A mechanism is provided in a data processing system for matching data to a dynamic set of signatures. The mechanism creates a state transition, result, and mask (STR&M) table based on a set of signatures. The mechanism executes scanner code in each of a plurality of hardware devices in parallel to form a plurality of matching engines. The mechanism loads the STR&M table into each of the plurality of matching engines. Responsive to receiving a plurality of input records, the mechanism distributes input strings from the plurality of input records among the plurality of matching engines and receiving signature set match results from the plurality of matching engines.