Database Load Engine for Multithreaded Customer Data Ingestion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for loading large volumes of customer data into operational databases are inefficient, often causing server slowdowns and disrupting normal database operations, especially during data merging or replication processes.
Innovation Solution
A load engine that validates, transforms, and partitions customer data according to established business rules and data models, allowing for multithreaded loading into target databases without disrupting ongoing operations, enabling efficient processing of millions of records while maintaining full database functionality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional server processes are used to add large quantities of data to the database, then data loading can be performed, but server runtime processing increases and normal database services are impacted
Solution Approach 1:
The patent segments the data loading process into distinct phases: data extraction from source systems, data transformation and validation, and data loading into the target database. This segmentation allows each phase to be optimized independently and enables parallel processing, thereby increasing data loading speed without impacting normal database services during operational hours.
Solution Approach 2:
The patent implements preliminary data validation and transformation before actual data loading into the operational database. By pre-processing and validating data in advance, the system ensures data quality and compatibility without requiring extensive runtime processing during normal database operations, thus maintaining service availability while enabling efficient data loading.
2Adaptability or versatility
If data transformation is performed to match target database format during merging, then data compatibility is improved, but server runtime processing increases
Solution Approach 1:
The patent performs data transformation and format conversion as a preliminary action before data loading into the target database. By pre-transforming data from disparate source formats into the target database schema in advance, the system achieves data compatibility without requiring extensive runtime processing during normal operations, thus reducing server runtime while maintaining adaptability.
Solution Approach 2:
The patent introduces an intermediary data transformation layer that acts as a mediator between source systems with different data formats and the target database. This intermediary layer standardizes data formats and validates data quality centrally, enabling data from multiple sources to be compatible with the target database while minimizing the runtime processing impact on the server.
3Quantity of substance
If large volumes of customer data are loaded directly to operational database, then data consolidation is achieved, but database operations are disrupted
Solution Approach 1:
The patent segments the data loading process into background batch processing and foreground operational queries. Large volumes of customer data are loaded during off-peak hours or in background processes, while normal database operations continue uninterrupted during business hours. This segmentation maintains database operational continuity while achieving data consolidation.
Solution Approach 2:
The patent performs data loading as a preliminary batch process during off-peak hours before operational traffic begins. By pre-loading large volumes of customer data during low-activity periods, the system consolidates data without disrupting normal database operations during business hours, thus maintaining ease of operation while achieving data volume consolidation.
Data Source
AI summary
The invention described herein provides a load engine and method for efficiently accomplishing mass conversions of customer data into an existing customer database. In particular, the method incorporates existing business rules for validating new customer data and for creating tables for the new customer data, creates load files for the new customer data, and provides a means for running multi-threaded data loads of the new customer data tables onto an existing customer database.