Metadata Validation Tool for Sequential Data Files
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data errors in datasets, such as keystroke errors or packet loss, can render data processing impossible and increase processing and memory usage due to the need for reprocessing, highlighting the need for effective metadata validation to ensure data integrity.
Innovation Solution
A metadata validation tool comprising a retrieval engine, extraction engine, preparation engine, and export engine that extracts metadata, determines data type and format requirements, generates validation routines, and produces error reports to identify and correct data errors, ensuring data integrity by reducing errors and improving processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored and processed without validation, then processing speed is maintained, but data errors increase leading to reprocessing and increased memory usage
Solution Approach 1:
The patent applies preliminary action by performing metadata extraction and validation routines before actual data processing occurs. The system extracts metadata from sequential data files, generates validation routines based on this metadata, and executes these routines to identify and correct errors before the main processing workflow begins, thereby preventing error propagation and reprocessing needs
2Reliability
If metadata validation is performed on all data entries, then data integrity is improved, but processing time and computational resources increase
Solution Approach 1:
The patent applies the extraction principle by separating metadata extraction and validation from the main data processing workflow. The system extracts only the necessary metadata information (data types, formats, validation rules) from sequential data files and stores this separately, then uses this extracted metadata to guide validation without processing the entire dataset repeatedly. This allows validation to be performed efficiently on extracted metadata rather than on all raw data entries
Solution Approach 2:
The system performs metadata extraction and validation rule generation in advance before the main processing occurs. By preparing validation routines preliminarily based on extracted metadata, the system avoids the need to perform complex validation logic during the main data processing phase, thereby reducing the time penalty of validation
3Reliability
If error validation and reporting is implemented, then data quality is improved, but device complexity increases
Solution Approach 1:
The patent applies segmentation by dividing the validation tool into distinct functional engines: a retrieval engine for obtaining sequential data files, an extraction engine for obtaining metadata and generating validation routines, and an export engine for producing error reports. This segmentation allows each component to perform a specific function, making the overall complex validation process manageable and maintainable through modular design
Data Source
AI summary
A metadata validation tool includes a retrieval engine, an extraction engine, a preparation engine a validation engine, and an export engine. The retrieval engine receives a sequential data file comprising a data table, the data table comprising a plurality of rows and a plurality of columns and the data table further comprising a plurality of data entries, wherein each data entry is placed in the data table at the intersection of one of the plurality of rows and one of the plurality of columns. The extraction engine extracts metadata from the sequential data file for at least one of the plurality of columns of data, wherein the metadata identifies the data table and a plurality of metadata identifiers. The extraction engine determines a metadata identifier for the extracted metadata, wherein the metadata identifier identifies a data field indicating a data type. The extraction engine determines, using the metadata identifier, a general requirement for at least one of the columns of data, the general requirement indicating the required format for each data entry in the at least one of the columns. The preparation engine generates a routine, the routine based on at least the general requirement, wherein the routine facilitates determining whether one or more of the plurality of data entries complies with the general requirement. The validation performs the routine and determines whether one or more rows of the plurality of rows comprises an error, the error indicating that at least one of the data entries of the one or more rows does not comply with the general requirement. Upon a determination that one or more rows of data comprise an error, the export engine generates a consolidated error report, the error report identifying an input row number indicating the one or more rows that contain the error.