Hybrid Unicode String Weighting for ASCII-Heavy Text Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Unicode text processing is costly and inefficient, especially in database operations that predominantly handle ASCII-based text, as existing methods require costly Unicode weighting for all characters, leading to performance issues in operations like hash joins and equality comparisons.
Innovation Solution
A hybrid approach that analyzes text strings to determine if they consist entirely of ASCII characters, using ASCII weights for pure ASCII strings and falling back to Unicode weighting only when necessary, with options for accent-insensitive or accent-sensitive comparisons, and using bitwise operations and Unicode weight tables for efficient processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If Unicode weighting is applied to all characters, then accurate text comparison is achieved, but processing performance deteriorates
Solution Approach 1:
The text string is segmented into ASCII and non-ASCII portions. The patent applies different weighting methods to different segments: fast ASCII-based weighting for ASCII characters and slower Unicode weighting only for non-ASCII characters. This segmentation resolves the contradiction by applying the expensive Unicode weighting only where necessary while using efficient ASCII weighting for the majority of characters.
Solution Approach 2:
Different quality levels of processing are applied to different parts of the text. ASCII characters receive simple, fast processing while non-ASCII characters receive more complex Unicode processing. This local differentiation maintains overall accuracy while improving performance by avoiding unnecessary complex processing for ASCII characters.
2Adaptability or versatility
If Unicode processing is used for all text, then comprehensive character support is achieved, but processing cost increases
Solution Approach 1:
The patent applies Unicode processing partially - only to the extent necessary for non-ASCII characters. ASCII characters are processed using efficient ASCII-specific rules, while Unicode processing is applied only to non-ASCII portions. This partial application maintains comprehensive character support while minimizing processing costs.
Solution Approach 2:
The processing parameters change based on the character type. For ASCII characters (code points 0-127), simple ASCII weighting rules are used. For non-ASCII characters, the patent switches to Unicode weighting parameters. This parameter change resolves the contradiction by adapting the processing intensity to the actual needs of each character.
Data Source
AI summary
A method compares text strings having Unicode encoding. The method receives a first string S=s1 s2 . . . sn and a second string T=t1 t2 . . . tm, where s1, s2, . . . , sn and t1, t2, . . . , tm are Unicode characters. The method computes a first string weight for the first string S according to a weight function ƒ. When S consists of ASCII characters, ƒ(S)=S. When S consists of ASCII characters and some accented ASCII characters that are replaceable by ASCII characters, ƒ(S)=g(s1) g(s2) . . . g(sn), where g(si)=si when si is an ASCII character and g(si)=si′ when si is an accented ASCII character that is replaceable by the corresponding ASCII character si′. When S includes one or more non-replaceable non-ASCII characters, the first string weight concatenates an ASCII weight prefix ƒA (S) and a Unicode weight suffix ƒU(S). The method also computes a second string weight for the second text string T. Equality of the strings is tested using the string weights.