Full Text Indexing Engine Diacritic Normalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing indexing systems face complications when dealing with files containing non-English text and symbols, as they struggle to efficiently create and search full text indexes that account for diacritic symbols and special characters, leading to ineffective searching capabilities.
Innovation Solution
An indexing method that normalizes input files by stripping diacritic symbols and storing words in a data structure, allowing a query engine to search for both stripped and original words with diacritics, and includes tables to manage variations such as casing and length, enabling efficient searching of English and non-English text.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If diacritic symbols are preserved in the index, then search accuracy for non-English text is improved, but indexing complexity and search time increase
Solution Approach 1:
The patent segments the indexing process into two independent data structures: one preserving diacritic symbols and another storing normalized versions without diacritics. This segmentation allows the system to handle both exact matches and fuzzy matches efficiently, reducing indexing complexity while maintaining search accuracy for non-English text.
Solution Approach 2:
The patent applies parameter changes by creating normalized versions of words with diacritic symbols removed or replaced. This transformation enables the search engine to handle multiple language scripts uniformly, simplifying the indexing structure while preserving the ability to perform accurate searches across different languages.
2Adaptability or versatility
If multiple variations of words are indexed, then search coverage is improved, but indexing time and storage requirements increase
Solution Approach 1:
The patent performs preliminary normalization during the indexing phase by storing both original and normalized forms of words. This preliminary action ensures that when queries are executed, the search engine can quickly match queries against pre-processed data without performing complex real-time transformations, thus improving indexing speed while maintaining comprehensive search coverage.
Solution Approach 2:
The patent creates simplified copies of words with diacritic symbols removed and stores them alongside original forms. These copied versions enable fast matching for queries that may or may not include diacritics, expanding search coverage without requiring the system to process every possible variation during indexing, thereby maintaining indexing efficiency.
3Measurement precision
If case variations are handled explicitly, then search accuracy is improved, but query processing time increases
Solution Approach 1:
The patent applies homogeneity by converting all text to a normalized form (lowercase without diacritics) during indexing. This creates a uniform representation that eliminates the need for case-sensitive comparisons during query processing, maintaining search accuracy while significantly reducing query processing time through consistent data structure.
Data Source
AI summary
An indexing engine generates a full text index of English and non-English files provided to the indexing engine. The indexing engine receives an input file for indexing, and normalizes the unique words contained in the input file. The normalizing includes stripping the words of any diacritical marks, taking into account different multilingual issues, case folding the words into lowercase, and the like. The normalized words are stored in a dictionary, and a word record is generated for each stored word. Each word record includes a flag that indicates whether one or more variations exist in the input file for the normalized word. One or more tables store information on the variations for the normalized words. When a query engine is invoked to search for an input query word, the variations are searched only if the user has set an option to consider such variations.