Transaction Data Parsing with Masked Language Model
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in parsing and normalizing user account data, particularly transaction descriptions, due to the lack of standardized formats across different entities, leading to messy and inconsistent data that is difficult to interpret and utilize effectively.
Innovation Solution
The implementation of a system that uses a masked language model and bidirectional parser, employing techniques like fuzzy string matching, transformer encoders, and linear-chain CRFs to extract and normalize location and entity information from transaction descriptions, generating standardized transaction data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional data parsing methods are used for transaction descriptions, then the system is simpler and faster, but the data extraction accuracy and normalization quality deteriorate due to lack of standardized formats
Solution Approach 1:
The parsing system is divided into multiple specialized modules: location extraction module, entity extraction module, and normalization module. Each module handles specific aspects of the parsing task independently, allowing for improved accuracy in extracting location and entity information while maintaining manageable system complexity through modular architecture
Solution Approach 2:
A masked language model serves as an intermediary component that processes raw transaction descriptions and generates structured representations. This intermediary layer bridges the gap between unstructured input data and the required structured output, improving extraction accuracy without requiring direct complex parsing logic
2Productivity
If manual data parsing and normalization is performed, then data accuracy can be maintained, but processing time and labor requirements increase significantly
Solution Approach 1:
The system employs automated parsing and normalization processes that operate independently without human intervention. The masked language model and bidirectional parser automatically extract, identify, and normalize location and entity information from transaction descriptions, enabling high-speed processing while maintaining accuracy through algorithmic rather than manual operations
Solution Approach 2:
Manual data parsing operations are replaced with automated computational processes using neural networks and parsing algorithms. The mechanical process of human reading and interpreting transaction descriptions is substituted with electronic processing through the masked language model and bidirectional parser, dramatically increasing processing speed
3Measurement precision
If fuzzy string matching and masked language models are applied, then entity recognition accuracy improves, but computational resources and processing time increase
Solution Approach 1:
The system applies fuzzy string matching and masked language models selectively rather than universally. These computationally intensive methods are applied only when necessary to achieve accurate entity recognition, while simpler parsing methods are used for straightforward cases, optimizing the balance between recognition accuracy and computational resource consumption
Data Source
AI summary
Systems and methods for data parsing are disclosed. In one aspect, a method of parsing raw data associated with one or more transactions involves receiving a text string including raw data for a transaction, matching the text string to a plurality of locations within a location corpus to extract location information from the text string, and identifying a candidate entity from the text string based on a similarity score with respect to a plurality of entities within an entity corpus. The method further involves in response to the similarity score of the identified candidate entity being less than a threshold score, generating entity information using the tokens indicative of entity information, and generating normalized transaction data including the extracted location information and one of the identified candidate entity or the generated entity information.