Pattern Matching Data Storage Using Dual Memory and Hash Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing pattern matching technologies face challenges in real-time processing due to software-based methods' inefficiencies and hardware-based methods' limitations, such as high resource costs and slow data access, especially when dealing with large volumes of data and varying pattern lengths in applications like intrusion detection and spam mail filtering.
Innovation Solution
A dual memory structure is implemented, using internal memory for fast access and external memory as a perfect hash table to store and compare patterns, with data segmented and hashed to minimize collisions and optimize storage and retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If software-based pattern matching methods are used, then implementation is simple and adaptable to various patterns, but processing speed is too slow for real-time applications
Solution Approach 1:
The patent replaces software-based pattern matching with a hardware-based system that uses dedicated circuitry and memory structures (internal memory, external memory, hash tables) to perform pattern matching operations at hardware speed, enabling real-time processing while maintaining adaptability through programmable components
Solution Approach 2:
The patent segments the pattern matching system into multiple components including internal memory for frequently accessed patterns, external memory for large-scale storage, hash tables for quick lookup, and dedicated processing units, allowing parallel operation and optimized performance for different types of pattern matching tasks
2Productivity
If hardware-based pattern matching is implemented to improve speed, then real-time processing capability is achieved, but resource cost and device complexity increase significantly
Solution Approach 1:
The patent applies local quality by assigning different storage media to different types of data based on access frequency and size requirements: internal memory stores frequently accessed small patterns for fastest access, external memory stores large-scale pattern databases, and hash tables store index information, optimizing both speed and resource utilization
Solution Approach 2:
The patent performs preliminary actions by pre-computing hash values for patterns and storing them in hash tables before actual pattern matching operations, allowing for extremely fast lookup and comparison during real-time processing without requiring complex runtime computations
3Quantity of substance
If external memory is used to store large numbers of patterns, then storage capacity is sufficient, but access speed becomes slower compared to internal memory
Solution Approach 1:
The patent segments the memory system into internal memory and external memory, each serving different purposes: internal memory provides fast access for actively used patterns, while external memory provides large-capacity storage for the complete pattern database, with a hash table structure enabling efficient indexing and retrieval between the two levels
Solution Approach 2:
The patent introduces hash tables as an intermediary structure between internal and external memory, storing compressed index information that enables quick location of patterns in external memory without requiring full-speed access to the entire external memory subsystem
4Adaptability or versatility
If the number of patterns to be compared increases to handle diverse applications, then pattern matching coverage improves, but hash collisions increase and retrieval time increases
Solution Approach 1:
The patent uses a nested structure where hash tables are embedded within the dual memory system, providing an additional layer of indexing that reduces hash collisions by distributing pattern references across multiple hash functions and table levels, thereby maintaining fast retrieval even as the number of patterns increases
Data Source
AI summary
A method and apparatus for storing pattern matching data and a pattern matching method using the method and apparatus are provided. The method of storing original data for pattern matching in a pattern matching apparatus includes: dividing the original data into segments of a predetermined size; performing a hash operation on each of the divided segments; determining whether or not the hash operation value of each segment causes a hash collision with a hash operation value stored in a first external memory disposed outside the pattern matching apparatus; and controlling the hash operation value of each segment determined not to cause a hash collision to be stored in the first external memory. According to the method and apparatus, the original data desired to be used for pattern matching can be stored at a faster speed in a pattern matching data storing apparatus.