Unicode Regex Filter Using Symbol Byte-Length Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hardware regular expression filters are inefficient for variable-length encoded data, such as Unicode, due to complexity and the need to handle various encoding schemes and variants.
Innovation Solution
The development of hardware regular expression filters that determine the symbol byte lengths of variable-length encoded symbols, allowing for efficient matching by comparing these lengths to a trigger, without needing to identify actual symbols or interpret graph encoding types.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If hardware regular expression filters are designed for ASCII encoded data with fixed-length encoding, then the filter implementation is straightforward and efficient, but the filter cannot handle variable-length encoded data such as Unicode
Solution Approach 1:
The patent changes the operating parameters of the hardware filter from fixed-length byte comparison to variable-length symbol comparison. The filter now operates on Unicode symbols rather than individual bytes, adapting its comparison logic to handle 1-4 byte code units according to UTF-8 encoding rules. This allows the same hardware structure to efficiently filter both ASCII and Unicode data by changing how input data is interpreted and compared.
2Productivity
If hardware regular expression filters process variable-length encoded data by comparing individual bytes, then data transfer efficiency improves, but the filter complexity increases significantly to handle various UTF variants and graph encodings
Solution Approach 1:
The patent extracts the essential filtering function from complex Unicode interpretation logic. Instead of implementing handlers for all UTF variants and graph encodings (NFD, NFC, NFKD, NFKC), the filter extracts only the necessary byte-length information from UTF-8 encoded symbols and performs comparison based on that. This extraction approach maintains high data transfer efficiency while avoiding the complexity of full Unicode compliance.
Solution Approach 2:
The patent segments the filtering process into two distinct stages: (1) a hardware filter stage that performs rapid byte-length comparison of UTF-8 encoded symbols to identify potential matches, and (2) a software processing stage that performs complete regular expression matching on the filtered results. This segmentation allows the hardware to handle the high-volume filtering task efficiently while the software handles the complex interpretation tasks.
3Measurement precision
If complete regular expression matching is performed on the host device, then matching accuracy is achieved, but data transfer between SSD and host device becomes inefficient and time consuming
Solution Approach 1:
The patent applies preliminary filtering action at the SSD hardware level before data transfer to the host. The hardware regular expression filter performs an initial screening of the data, identifying only those portions that potentially match the search pattern. This preliminary action significantly reduces the volume of data that needs to be transferred to the host for complete matching, thereby reducing data transfer time while preserving matching accuracy for the filtered results.
Data Source
AI summary
An apparatus is provided that includes a memory system that has a memory die that includes a circuit configured to receive at a first input terminal a plurality of bytes comprising an input stream of a first plurality of variable-length encoded symbols, and determine a sequence of symbol byte lengths for each of the first plurality of variable-length encoded symbols.