Database Load Error Tracking for Parallel Bulk Data Ingestion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, particularly when handling large volumes of data.
Innovation Solution
A parallelized database system architecture that includes a parallelized data input, store, retrieve, and process sub-system, along with a query and response system, utilizing a network of computing devices and nodes to distribute and optimize query execution across multiple processing core resources, enabling efficient storage and retrieval of massive data sets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored in a traditional database system, then data can be stored and retrieved, but processing speed is limited due to hardware constraints and sequential processing
Solution Approach 1:
The patent divides the database system into multiple independent nodes that can process data in parallel. Each node handles specific portions of the workload independently, eliminating the sequential processing bottleneck of traditional single-system databases while maintaining manageable complexity through modular architecture.
Solution Approach 2:
The patent transitions from a single-dimensional sequential processing model to a multi-dimensional parallel processing architecture. By distributing data across multiple nodes in a networked configuration, the system adds spatial and organizational dimensions to processing, enabling simultaneous operations across multiple processing cores.
2Productivity
If hardware resources are increased to improve processing speed, then more data can be processed, but hardware costs and physical constraints become limiting factors
Solution Approach 1:
The patent changes the fundamental parameter of processing architecture from sequential to parallel, and from centralized to distributed. This allows the system to achieve higher throughput not by adding more hardware resources linearly, but by fundamentally changing how existing resources are utilized across multiple nodes and processing cores.
Solution Approach 2:
Each node in the distributed system is designed to be multi-functional, capable of performing various database operations (storage, retrieval, processing) independently. This universality allows the system to scale horizontally by adding nodes that can immediately contribute to processing throughput without requiring specialized hardware additions.
3Productivity
If data is loaded in bulk without error tracking, then loading speed is faster, but data integrity cannot be ensured
Solution Approach 1:
The patent implements preliminary error tracking mechanisms during the data loading process. By continuously monitoring and tracking errors as data is loaded in bulk, the system can identify and handle errors in real-time without interrupting the high-speed loading process, thus maintaining both speed and reliability.
Solution Approach 2:
The system incorporates feedback loops during data loading that continuously monitor for errors and provide information back to the loading process. This feedback mechanism enables the system to maintain high loading speeds while simultaneously ensuring data integrity through real-time error detection and reporting.
Data Source
AI summary
A database system is operable to initiate performance of a loading process to load a plurality of records included across a plurality of files based on processing the plurality of files to process the plurality of records for storage via the database system. During performance of the loading process, load error tracking data is maintained in accordance with error handling configuration data. When a record-level error occurs during processing of a corresponding record of the plurality of records, the load error tracking data is updated to indicate the record-level error for the corresponding record. When a file-level error occurs during processing of a corresponding file of the plurality of files, the load error tracking data is updated to indicate the file-level error for the corresponding file. The load error tracking data is communicated for access by a user entity associated with the loading process.