Data Ingestion System with Compliance and Deduplication Modules
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems face challenges in efficiently integrating, validating, and updating data objects, particularly in ensuring compliance and deduplication across diverse data types and sources, which affects data integrity and accuracy.
Innovation Solution
The implementation of an ingestion system comprising an ingestion module, a compliance module, and a deduplication module that processes and validates data objects by parsing, comparing them against defined properties, identifying non-compliant parts, and generating compliance and deduplication alerts, while also managing duplicate data through a deduplication database.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data objects are integrated from diverse sources without comprehensive validation, then integration speed is improved, but data integrity and accuracy deteriorate
Solution Approach 1:
The system performs preliminary validation and compliance checking on data objects before they are fully integrated into the database. The compliance module validates data against schemas and business rules during the ingestion process, preventing invalid data from compromising integrity while maintaining integration throughput.
Solution Approach 2:
The patent introduces intermediary modules (compliance module, deduplication module) between the data source and the final database storage. These intermediaries process and validate data objects, ensuring data integrity without becoming a bottleneck to integration speed.
2Measurement precision
If comprehensive compliance validation is performed on all data parts, then data accuracy is improved, but processing time increases
Solution Approach 1:
The compliance validation process is segmented into multiple independent checks (schema validation, business rule validation, deduplication checks) that can be performed in parallel on different parts of the data object, reducing overall processing time while maintaining comprehensive validation.
Solution Approach 2:
The system performs validation on critical data parts with higher priority, and optional or less strict validation on non-critical parts. This partial action approach ensures data accuracy for essential fields while reducing processing overhead for less important data elements.
3Reliability
If duplicate detection is performed across all data objects, then data quality is improved, but system complexity increases
Solution Approach 1:
The system creates simplified copies or hashes of data objects for comparison purposes, rather than performing complex full-data comparisons. This copying approach enables efficient duplicate detection with reduced computational complexity.
Solution Approach 2:
The deduplication process uses parameter changes such as converting data to hash values or normalized forms for comparison. This transformation simplifies the complexity of comparing entire data objects while maintaining the ability to detect duplicates accurately.
4Reliability
If multiple validation modules are implemented, then compliance assurance is improved, but processing overhead increases
Solution Approach 1:
Multiple validation checks are merged into a single compliance module that performs schema validation, business rule validation, and deduplication in an integrated manner. This merging reduces processing overhead by eliminating redundant operations and shared code paths while maintaining comprehensive compliance assurance.
Data Source
AI summary
Systems and methods are provided to ingest data objects from a flat file server for use in one or more system operations including providing a renderable data object to a user and updating a data item database. As described, the ingestion system includes an ingestion module, a flat file module, a compliance module, and a deduplication module wherein the modules together ingest a flat file data object, parse and process a renderable data object from the flat file data object, and store the renderable data object in a renderable object database.