Single-Pass Data Record Cleansing and Linking System
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional software systems for handling large databases are inefficient as they perform operations one at a time, leading to excessive reads and writes from remote storage, increased setup time, and high processing costs, and fail to recycle data records through previous processes when changes occur, resulting in errors and increased time consumption.
Innovation Solution
A system and method that reads a data record from an input file once, processes it against multiple reference files, and writes the results to a target file, allowing for recycling of data records through previous processes if changes occur, thereby reducing the need for multiple reads and writes to remote storage and improving processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional software systems perform operations one at a time on large databases, then each operation can be performed accurately, but the processing time and machine resources increase significantly
Solution Approach 1:
The patent combines multiple database operations (reading, processing, writing) into a single integrated pass through the data records. Instead of performing operations sequentially with multiple reads and writes, the system processes all operations in one traversal of the data, merging multiple steps into a unified processing flow that maintains accuracy while dramatically improving speed.
Solution Approach 2:
The patent performs preliminary actions by reading and processing data records in advance during the single pass through the database. By preparing and processing all necessary operations during this initial traversal, the system eliminates the need for subsequent repeated reads and writes, thereby reducing overall processing time while maintaining operational accuracy.
2Adaptability or versatility
If conventional systems read and write data records multiple times for different operations, then each operation can access the data, but the number of reads and writes increases excessively
Solution Approach 1:
The patent implements a universal processing mechanism that handles multiple operations (reading, processing, writing) within a single pass through the data records. This multi-functional approach allows the system to perform various operations without requiring separate reads and writes for each operation, thereby reducing time loss while maintaining operational flexibility.
Solution Approach 2:
The patent maintains continuous useful action by processing data records in an uninterrupted single pass through the database. Instead of interrupting the processing flow for separate reads and writes, the system continues processing operations sequentially within one traversal, eliminating idle time and reducing overall processing duration while preserving the ability to perform multiple operations.
3Device complexity
If conventional systems do not recycle data records through previous processes, then the process flow is simple, but errors occur and time consumption increases when changes are needed
Solution Approach 1:
The patent implements feedback mechanisms that allow data records to be recycled back through previous processes when changes are detected. This feedback loop monitors changes in data records and redirects them through prior processing steps to ensure consistency and accuracy, thereby improving reliability while managing complexity through targeted recycling rather than complete reprocessing.
Solution Approach 2:
The patent introduces dynamic behavior to the processing flow by enabling data records to be recycled through previous processes based on detected changes. This dynamic approach allows the system to adapt its processing path in real-time, recycling records only when necessary to maintain accuracy, thereby balancing complexity with improved reliability through conditional reprocessing.
4Ease of operation
If conventional systems process data records sequentially one at a time, then each record can be processed independently, but the total processing time increases significantly
Solution Approach 1:
The patent segments the processing operations into distinct functional steps within a single pass through the data records. By dividing the processing into manageable segments (reading, processing, writing) that can be executed sequentially within one traversal, the system maintains operational simplicity while eliminating the time loss associated with multiple separate passes.
Data Source
AI summary
A system and method for reading a data record from an input file only once, processing that data record according to one or more references files, and then writing out the cleansed and updated data record to a target file such that the data record is read and written to remote storage only once, thereby making a single pass through a given database of data records. Each data record (comprising of multiple data elements) of the input file is reviewed, verified, and corrected against one or more reference databases containing similar information, assigned a unique identifying key, and, optionally, appended with new additional data elements of a matching data record from a new-data database.