Machine-Learned Field-Specific Tokenization for Customer Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing customer data management systems face challenges in accurately tokenizing and standardizing customer data due to variations in formats and languages, leading to incorrect or duplicate records, especially in multi-tenant databases with diverse data sources.
Innovation Solution
A machine-learning based field-specific tokenization method that creates a training set by replacing characters with classes and associating labels to determine token boundaries, allowing for accurate identification of tokens in customer data fields, such as email addresses and street addresses, without relying on human intervention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional rule-based tokenization methods are used, then the system is simple to implement, but the accuracy of tokenization decreases due to variations in formats and languages
Solution Approach 1:
The patent replaces traditional rule-based mechanical tokenization systems with a machine learning model that automatically learns tokenization patterns from training data. The model substitutes manual rule configuration with automated pattern recognition, achieving higher accuracy across diverse formats and languages without requiring explicit programming of tokenization rules for each case.
Solution Approach 2:
The system changes the parameters of tokenization by using a trained machine learning model that adapts to different data formats, languages, and field types. Instead of fixed rules, the model dynamically adjusts tokenization behavior based on learned patterns from training data, allowing accurate handling of variations in customer data formats across different sources and languages.
2Adaptability or versatility
If manual tokenization rules are created for each data source, then the tokenization can be customized, but the time and effort required increases significantly
Solution Approach 1:
The machine learning model performs self-service by automatically learning tokenization patterns from training data representing multiple data sources and formats. The system configures itself through the training process, eliminating the need for manual rule creation for each data source. The model adapts to different formats, languages, and field types autonomously during training, then applies learned patterns to new data without additional configuration.
Solution Approach 2:
The trained machine learning model achieves universality by handling multiple data sources, formats, languages, and field types with a single system. Instead of requiring separate tokenization rules for each data source, the model learns generalized patterns that apply across diverse customer data, enabling one system to serve multiple customization needs without proportional increases in configuration effort.
3Measurement precision
If standard tokenization methods are used without field-specific training, then the processing is faster, but the accuracy for specific fields like email addresses and street addresses decreases
Solution Approach 1:
The patent applies local quality by training the machine learning model with field-specific characteristics for different customer data types. Each field (email addresses, street addresses, phone numbers) receives specialized training attention, allowing the model to learn and apply field-specific tokenization patterns. This targeted approach improves accuracy for specific fields while maintaining overall processing efficiency through the model's learned generalizations.
Data Source
AI summary
A training set is created via creating adjacent classified substrings by using character classes to replace corresponding characters in adjacent substrings in each training character string, and associating each pair of adjacent classified substrings and each pair of adjacent substrings with corresponding labels indicating whether corresponding pairs include any token boundary. The system splits input character string into beginning and ending parts and creates classified beginning part by replacing beginning part character with corresponding class and classified ending part by replacing ending part character with corresponding class. The machine-learning model determines probability of token identification, based on training set to determine count of instances that classified beginning part is paired with classified ending part and count of corresponding labels that indicate inclusion of any token boundary. If token identification probability satisfies threshold, the system identifies beginning part as token and ending part as remainder of input character string.