Tokenizing User-Annotated Names via Frequency Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in accurately searching and categorizing metadata tags due to differences in capitalization and spacing conventions, leading to inconsistent tokenization across users and organizations.
Innovation Solution
A method for tokenizing user-annotated names involves creating a custom dictionary from an example set of metadata, assigning frequency scores to keywords, and performing semantic analysis to split additional names into tokens based on top combined frequency scores, using a computing device to handle arbitrary structures and conventions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional substring matching procedures are used, then simple matching can be performed, but accurate search and categorization fails when metadata tags use different capitalization and spacing conventions
Solution Approach 1:
The patent segments the metadata tag into multiple tokens by splitting on whitespace and punctuation, then normalizes each token for comparison. This allows the system to handle different capitalization and spacing conventions by treating 'My Document' and 'my document' as equivalent token sets, resolving the contradiction between precise matching and convention flexibility.
Solution Approach 2:
The patent changes the comparison parameters by normalizing text case (converting to lowercase) and ignoring punctuation during matching. This parameter transformation enables accurate search across different capitalization and spacing conventions without requiring exact string matches, simultaneously improving search accuracy and adaptability.
2Stability of the object's composition
If fixed spacing and capitalization rules are enforced, then consistent tokenization can be achieved, but the system cannot handle arbitrary user conventions
Solution Approach 1:
The patent implements dynamic tokenization by detecting the presence of delimiters (spaces, tabs, punctuation) and adjusting the tokenization strategy accordingly. When delimiters are present, the system splits on those delimiters; when absent, it uses alternative splitting rules. This dynamic approach maintains consistency while adapting to arbitrary user conventions.
Solution Approach 2:
The patent introduces a normalization layer as an intermediary between the raw metadata tag and the tokenization process. This intermediary standardizes the input by normalizing case and handling punctuation, then passes the normalized form to the tokenization algorithm. This mediator enables consistent tokenization across diverse conventions without requiring the core tokenization logic to handle every variation directly.
3Measurement precision
If custom dictionary with frequency scoring is created, then accurate tokenization can be achieved, but processing time increases
Solution Approach 1:
The patent performs preliminary action by pre-computing and storing frequency scores for common words and phrases in a dictionary data structure. During tokenization, the system queries this pre-computed dictionary rather than calculating frequencies in real-time. This preliminary preparation enables accurate frequency-based tokenization without the time cost of on-demand frequency calculation.
Solution Approach 2:
The patent creates a simplified copy of the frequency scoring mechanism using a dictionary data structure that stores pre-computed scores. Instead of performing complex frequency analysis during each tokenization operation, the system uses this copied frequency information from the dictionary, dramatically reducing processing time while maintaining tokenization accuracy.
Data Source
AI summary
A disclosed computer-implemented method for tokenizing user-annotated names may include (1) identifying an example set of user-annotated names, (2) creating a custom dictionary that includes known keywords by (a) extracting a set of known keywords from the example set of user-annotated names and (b) assigning a frequency score to each known keyword in the set of known keywords based on the respective frequency of each known keyword within the example set, and (3) enabling the computing device to tokenize an additional user-annotated name of arbitrary structure by performing a semantic analysis including (a) assigning, using the custom dictionary, a frequency score to a substring of the additional user-annotated name based on the substring matching the known keyword and (b) splitting the additional user-annotated name into tokens according to a permutation of substrings that received a top combined frequency score. Various other methods, systems, and computer-readable media are also disclosed.