Real-time Transaction Data Correction via Semantic Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Inaccuracies in data input, particularly in conforming to defined formats and taxonomies, lead to inhibited processing and resource-intensive audits in systems like enterprise resource planning (ERP) systems, especially during data entry in general ledgers, where hundreds of thousands of transactions require accurate cost category assignments.
Innovation Solution
Implementing real-time data input correction through word embeddings and clustering, using semantic search to automatically assign categories and provide reasoning for selection, thereby facilitating accurate data entry and reducing audit burdens by processing transactions at their point of origin.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual data entry is performed without real-time correction, then data input speed is maintained, but data accuracy and format conformity deteriorate
Solution Approach 1:
The system performs preliminary actions by providing real-time suggestions and corrections during data entry. Word embeddings are generated and clustered beforehand to enable rapid matching, and the system proactively suggests corrections before the user finalizes incorrect data, thus improving accuracy without significantly slowing down input speed
Solution Approach 2:
The system implements continuous feedback during data entry by comparing input data against clustered word embeddings and providing real-time suggestions. This feedback loop allows users to correct errors immediately, improving data accuracy while maintaining productivity through automated assistance rather than manual review
2Measurement precision
If real-time semantic processing is implemented, then data accuracy and category assignment quality improve, but computational resource consumption increases
Solution Approach 1:
The system performs preliminary processing by pre-computing word embeddings and organizing them into clustered categories before actual data entry. This preprocessing step enables rapid semantic matching during real-time operation, improving category assignment accuracy while reducing the computational burden during active use
Solution Approach 2:
The computational task is segmented into distinct phases: offline embedding generation and clustering, and online semantic matching. By separating these computationally intensive tasks, the system achieves high accuracy in category assignment while managing resource consumption through efficient offline preprocessing
3Reliability
If comprehensive audit processes are executed to correct data non-conformities, then data quality improves, but processing time and resource expenditure increase
Solution Approach 1:
The system performs data quality assurance preliminarily during the data entry process itself. By providing real-time suggestions and corrections at the point of input, the system prevents non-conformities before they occur, eliminating the need for subsequent audit processes and thereby improving data quality without time loss
Solution Approach 2:
The system converts the potential harm of data entry errors into a benefit by using semantic analysis to proactively identify and suggest corrections. What would otherwise require time-consuming audits is transformed into immediate automated assistance during entry, maintaining high data quality while eliminating audit delays
Data Source
AI summary
Implementations are directed to providing categorization of transactional data, and include actions of providing a plurality of word embeddings based on domain-relevant text data, clustering word embeddings of the plurality of word embeddings into a plurality of clusters, receiving, in real-time, transactional data representative of a transaction, providing a category that is to be assigned to the transaction based on the transactional data, and the plurality of clusters, processing the category, the transactional data, the text data, and the plurality of clusters using a semantic search to provide reason text data, the reason text data representing a reason for selection of the category assigned to the transaction, and storing the transaction data, the category, and the reason text data in a transaction database.