Regex Token Abstraction for Data Normalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data normalization techniques require manual coding of regular expressions by skilled developers, which is time-consuming and labor-intensive, and fail to efficiently handle variations in input data formats, leading to delays and increased costs.
Innovation Solution
The use of regex tokens and token rules, generated in part using machine learning techniques, provides an abstraction layer that simplifies data normalization by allowing users to create and modify rules without direct manipulation of regular expressions, enabling easier handling of data variations and improved system performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual coding of regular expressions by skilled developers is used, then data normalization can be performed, but the process is time-consuming and labor-intensive
Solution Approach 1:
The patent introduces an intermediary layer between the raw data and the normalization process. This intermediary consists of a vocabulary of tokens and a grammar system that automatically generates regular expressions. Instead of developers manually crafting complex regex patterns, the system uses this intermediary vocabulary and grammar to automatically produce the necessary normalization rules, significantly reducing development time while maintaining accuracy.
Solution Approach 2:
The system performs self-service by automatically generating regular expressions from the vocabulary and grammar without requiring manual intervention from skilled developers. The grammar system takes the vocabulary tokens and automatically composes regular expressions that match the desired normalization patterns, enabling the system to service itself rather than requiring human expertise for each normalization task.
2Reliability
If manual coding of regular expressions by skilled developers is used, then data normalization can be performed, but the process is labor-intensive and increases costs
Solution Approach 1:
The patent introduces an intermediary layer between the raw data and the normalization process. This intermediary consists of a vocabulary of tokens and a grammar system that automatically generates regular expressions. Instead of developers manually crafting complex regex patterns, the system uses this intermediary vocabulary and grammar to automatically produce the necessary normalization rules, significantly reducing development time while maintaining accuracy.
Solution Approach 2:
The patent replaces the mechanical system of manual regex coding with an automated grammatical system. Instead of developers mechanically writing and testing regular expressions, the system uses a grammar-based approach that automatically generates the necessary patterns from a vocabulary of tokens, substituting manual mechanical work with automated computational processes.
3Adaptability or versatility
If complex regular expressions are used to handle data variations, then normalization can be achieved, but the complexity increases and non-technical users cannot manage it
Solution Approach 1:
The patent segments the normalization system into distinct components: a vocabulary of atomic tokens, a grammar system for combining tokens, and a regular expression generator. This segmentation allows complex data variation handling to be achieved through composition of simple, manageable units rather than monolithic complex expressions, making the system more adaptable while reducing perceived complexity.
Solution Approach 2:
The patent introduces an intermediary layer between the raw data and the normalization process. This intermediary consists of a vocabulary of tokens and a grammar system that automatically generates regular expressions. Instead of developers manually crafting complex regex patterns, the system uses this intermediary vocabulary and grammar to automatically produce the necessary normalization rules, significantly reducing development time while maintaining accuracy.
4Manufacturing precision
If manual regular expression coding is required, then precise control over normalization is achieved, but skilled developers are needed which increases costs
Solution Approach 1:
The system performs self-service by automatically generating regular expressions from the vocabulary and grammar without requiring manual intervention from skilled developers. The grammar system takes the vocabulary tokens and automatically composes regular expressions that match the desired normalization patterns, enabling the system to service itself rather than requiring human expertise for each normalization task.
Solution Approach 2:
The patent replaces the mechanical system of manual regex coding with an automated grammatical system. Instead of developers mechanically writing and testing regular expressions, the system uses a grammar-based approach that automatically generates the necessary patterns from a vocabulary of tokens, substituting manual mechanical work with automated computational processes.
Data Source
AI summary
Described are techniques and systems to process input data into normalized data using regular expression (“regex”) tokens to build rules. The regex tokens refer to a predefined regex phrase. A developer or an automated system may create the regex tokens. A user or automated system may combine and reuse the regex tokens in various ways to express different normalization rules. These rules may be automatically processed to generate regexs. The regexs are used in processing the input data to generate the normalized data.