Data Extraction Validation Using Existence and Uniqueness Checks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The process of extracting data from operational systems to OLAP systems is prone to errors due to complex data structures and heterogeneous data formats, leading to issues of existence, uniqueness, and correctness, which reduces the utility and benefits of the extracted data.
Innovation Solution
A system and method for validating the data extraction process that includes a queue to store extracted data, a processor to determine the existence and uniqueness of identifiers, and a validation interface to report validation results, ensuring accurate and unique data transfer from operational systems to OLAP systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data extraction is performed from operational systems to OLAP systems, then data can be transferred and stored for analysis, but errors occur due to complex data structures and heterogeneous data formats
Solution Approach 1:
The patent applies preliminary action by validating data against extraction templates before the actual extraction process. The validation interface checks whether extracted data conforms to the expected structure and format defined in templates, allowing errors to be detected and corrected before data is loaded into the OLAP system. This prevents erroneous data from compromising system reliability while maintaining efficient extraction operations.
Solution Approach 2:
The patent implements feedback mechanisms where the validation interface provides real-time verification of extracted data against operational system data. The system compares extracted data with source data to ensure accuracy, and any discrepancies are identified and reported. This feedback loop ensures data integrity while maintaining high extraction productivity by allowing continuous validation without blocking the extraction process.
2Reliability
If validation of extracted data is performed, then data accuracy and uniqueness are improved, but system complexity increases
Solution Approach 1:
The patent uses an intermediary approach by introducing a validation interface that acts as a mediator between the extraction process and the OLAP system. This interface layer handles all validation logic separately, allowing the extraction process to remain simple while ensuring data accuracy through the validation layer. The validation interface compares extracted data against templates and operational system data, providing a clear separation of concerns that reduces overall system complexity.
Solution Approach 2:
The patent applies copying by creating virtual copies of data structures through extraction templates. Instead of directly manipulating complex data structures during validation, the system uses template copies that define the expected structure. The validation process compares extracted data against these template copies, simplifying the validation logic while maintaining high accuracy. This copying approach allows complex validation of heterogeneous data without increasing system complexity.
3Ease of manufacture
If data transformation and evaluation are performed, then data is prepared for OLAP system storage, but errors arise in interpreting complex data formats
Solution Approach 1:
The patent applies preliminary action by defining extraction templates that specify the expected data formats and structures before the extraction process begins. These templates are created based on the operational system's data structures, allowing the validation interface to check extracted data against the predefined templates. This ensures accurate interpretation of complex data formats while maintaining simple transformation processes, as the templates guide the entire extraction and validation workflow.
Solution Approach 2:
The patent implements feedback by continuously comparing extracted and transformed data against the operational system's actual data structures. The validation interface provides real-time feedback on format interpretation accuracy, identifying any deviations from the expected structure. This feedback mechanism ensures high manufacturing precision in data format interpretation while keeping the data preparation process simple, as the feedback guides corrections automatically.
Data Source
AI summary
Validation of an extraction process from an operation system to an on-line analytics and processing (“OLAP”) system may be achieved utilizing a function module that reads data from a queue and outputs the data in a structured form. A second function module may be used to perform an existence and uniqueness check on the data to determine the existence and/or uniqueness of various data elements.