Augmented Lookups Validate Missing Code Table Values
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data warehouse processing is disrupted by lookup misses in code tables due to outdated or missing data values, leading to costly re-processing and resource inefficiencies.
Innovation Solution
A method and system that automatically determines the validity of missing data values by searching for their frequency and occurrence in transaction messages, storing valid values in a secondary data storage, and updating the code table accordingly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If transaction messages with missing code table values are discarded or delayed for human evaluation, then data processing accuracy is maintained, but processing time and computational resources increase significantly
Solution Approach 1:
The system performs preliminary actions by automatically detecting missing code table values and proactively searching for their meanings in historical transaction messages before final processing. This preliminary search and validation prevents subsequent processing delays and ensures accurate handling of unknown codes without requiring human intervention.
Solution Approach 2:
The system implements feedback mechanisms by monitoring code table lookup results and using this information to automatically update code tables with newly discovered values. The feedback loop continuously refines the code table based on actual transaction data, improving future lookup accuracy and reducing the need for manual code table maintenance.
2Reliability
If code tables are manually updated with new codes, then data processing accuracy is maintained, but processing speed decreases due to delayed transmission and updates
Solution Approach 1:
The system performs self-service by automatically detecting unknown code values, searching for their meanings in historical data, and updating the code table without human intervention. This self-updating mechanism eliminates manual code table maintenance, significantly improving processing speed while maintaining accuracy through automated validation.
Solution Approach 2:
The system ensures continuous improvement of the code table by continuously monitoring transaction messages and automatically updating code table entries. This continuous operation eliminates gaps in code table updates, ensuring the system always has the most current and accurate code mappings available for processing.
3Productivity
If transaction messages with unknown codes are stored for later processing, then processing completeness is maintained, but system complexity and storage requirements increase
Solution Approach 1:
The system extracts and isolates the problem of unknown codes by creating a separate search and validation process that handles only the exceptional cases. This extraction allows the main processing flow to remain simple and efficient, while a dedicated mechanism handles the complexity of code table updates and unknown value resolution.
Data Source
AI summary
A method is disclosed. The method includes receiving a transaction message comprising a data value in a data field, determining a code table associated with the data field, and determining that the data value is not present within the code table. When the data value is not present within the code table, the method includes searching a first data storage for other transaction messages that comprise the data value. In response to searching, the method includes determining if a number of transaction messages with the data value in the first data storage exceeds a predetermined number and/or frequency. When the number of transactions messages with the data value in the first data storage exceeds the predetermined number and/or frequency, storing the transaction message comprising the data value in a second data storage.