Cloud Data Loading Component for Large File Partitioning

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing technologies face challenges in efficiently loading and processing large data files, such as financial data with millions of records, into cloud-based data stores, particularly in terms of data format conversion, validation, and error handling.

Innovation Solution

A user-customizable cloud loading component is integrated into an integrated development environment (IDE), which partitions large data files using a round robin approach, converts data into a JSON-based format with automatically generated schemas, validates the data, and performs reconciliation to ensure data integrity and error-free loading.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If large data files are loaded directly into cloud data stores, then data loading speed is improved, but data integrity and error handling deteriorate

Engineering Contradiction:

Improvedata loading speedVSAvoiddata integrity

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system divides large data files into smaller partitions before loading into the cloud data store. Each partition is processed independently with its own validation and error handling, allowing parallel loading while maintaining data integrity through granular control and reconciliation of individual partitions.

Inventive Principle:

Principle #1Segmentation

2Measurement precision

If data format conversion is performed manually, then data validation accuracy is improved, but processing time increases

Engineering Contradiction:

Improvedata validation accuracyVSAvoidprocessing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system automatically generates schemas and performs data format conversion without manual intervention. The conversion process includes built-in validation that checks data integrity during transformation, maintaining accuracy while eliminating the time cost of manual format conversion.

Inventive Principle:

Principle #25Self-service

3Reliability

If comprehensive error checking is performed, then data reliability is improved, but processing complexity increases

Engineering Contradiction:

Improvedata reliabilityVSAvoidprocessing complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system performs error checking and reconciliation at multiple stages before final data loading, including pre-loading validation and post-loading reconciliation. This preliminary action approach ensures data reliability is established early, simplifying the overall process by preventing errors rather than requiring complex post-processing error handling.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS20250199776A1System and method for customizable large data loading

Publication Date: 2025.06.19 WELLS FARGO BANK NA

AI summary

Aspects of the present disclosure include systems and methods for receiving as input a large data file, and partitioning the large data file into a plurality of smaller partitioned data files. The methods further include generating, for each partitioned data file, a data schema based on an automated analysis of each partitioned data file, and generating a control file for each partitioned data file containing a record count. The methods also include loading, via a cloud loading system, each partitioned data file into a data store external to the cloud loading system based on the data schema, and validating that the data store has received all records in each of the partitioned data files based on the control file, wherein the cloud loading system is provided as a user-configurable cloud loading component used for developing a computer program.