Locality-Sensitive Hashing for Text Log Normalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text normalization techniques are computationally expensive and resource-intensive, making them inefficient for large-scale datasets, especially when dealing with noisy textual data from sources like social media and search logs, which contain spelling errors and non-standard punctuation.
Innovation Solution
The use of locality-sensitive hashing (LSH) and graph structures to normalize textual data, allowing for efficient mapping of word variants to canonical forms without requiring explicit heuristics or supervised learning, and representing insertion, deletion, and replacement probabilities through edge weights in the graph structure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If traditional text normalization techniques (spelling correction, stemming, lemmatization) are used, then text cleaning quality is improved, but computational cost and processing time increase significantly
Solution Approach 1:
The patent segments the text normalization problem into character-level operations. Instead of treating entire words as units for correction, the system breaks down words into individual characters and applies localized transformations based on character n-grams and edit distances, enabling parallel processing and reducing computational complexity per word.
Solution Approach 2:
The patent changes the parameters of text normalization from word-level linguistic rules to character-level probabilistic transformations. By representing text as sequences of characters with associated probability distributions for insertion, deletion, substitution, and movement operations, the system enables efficient computation through matrix operations and dynamic programming rather than expensive rule-based processing.
2Manufacturing precision
If traditional text normalization techniques are applied to large-scale datasets, then text quality is improved, but resource consumption and processing time become prohibitively expensive
Solution Approach 1:
The patent performs preliminary action by pre-computing character-level probability matrices and edit distance metrics during an offline training phase. These pre-computed parameters are stored and reused during online text normalization, eliminating the need for expensive real-time computation and enabling rapid processing of large datasets with consistent quality.
Solution Approach 2:
The patent uses copying by creating and storing probability distribution tables and transformation matrices that can be replicated and applied to multiple text instances. Instead of re-computing normalization parameters for each text document, the system copies and applies pre-learned character-level transformation rules, dramatically reducing processing time while maintaining normalization quality.
3Reliability
If comprehensive text cleaning is performed on noisy real-world data, then data quality is improved, but computational complexity and resource requirements increase
Solution Approach 1:
The patent substitutes complex mechanical rule-based systems with a probabilistic model based on character-level operations. Instead of using intricate spelling correction rules, stemming algorithms, and lemmatization dictionaries, the system employs a unified probabilistic framework that models text transformations as random processes, simplifying the overall system architecture while handling diverse normalization scenarios.
Data Source
AI summary
Techniques for improved text normalization are provided. Signatures are generated for a first word and a second word using a locality-sensitive hashing technique. A graph is constructed based on the first and second signatures, by creating a first node in the graph for the first word, creating a second node in the graph for the second word, and creating an edge in the graph connecting the first and second nodes upon determining that the first and second signatures match. A mapping from the first word to the second word is then generated based on the graph.