Data Upload Tool Parallel Batch Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in efficiently processing and uploading large datasets across network nodes, leading to increased computer resource usage and potential errors, as they lack the ability to gather comprehensive analytics for entire data processing and uploading jobs, and generate reports for multiple users efficiently.
Innovation Solution
The proposed solution involves an upload tool that generates data tables, breaks them into batches for parallel processing and uploading, and uses analytics tools to determine network node performance, identifying and correcting errors, while also providing a reporting tool that generates a single report for multiple users, reducing processing power and memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is transmitted within a network environment to perform ETL jobs, then data processing capability is improved, but computer resource usage (memory, storage, network bandwidth, CPU) increases
Solution Approach 1:
The patent segments large datasets into smaller batches for processing. The ETL process is divided into multiple stages (extract, transform, load) that can be executed in parallel across different network nodes. This segmentation allows data processing to be distributed, improving overall capability while reducing the resource burden on any single node or operation.
2Adaptability or versatility
If data tables are generated from multiple files with authentication, then data organization capability is improved, but processing time and complexity increase
Solution Approach 1:
The patent implements preliminary authentication and metadata extraction when files are first uploaded to the network node. Authentication credentials are verified in advance, and file metadata (structure, format, size) is pre-processed and stored. This preliminary action eliminates the need for repeated authentication and analysis during data table generation, significantly reducing processing time while maintaining strong organizational capabilities.
3Quantity of substance
If large data tables are uploaded to network nodes, then data storage capability is improved, but error rates and upload failures increase
Solution Approach 1:
The patent divides large data tables into smaller batches or chunks for upload. Each batch is independently transmitted to the network node and acknowledged separately. If an upload fails, only the specific failed batch needs to be retransmitted rather than the entire data table. This segmentation approach maintains high data storage capability while significantly improving upload reliability and reducing error propagation.
4Adaptability or versatility
If separate data reports are generated for each user, then user-specific customization is improved, but processing usage and memory usage increase
Solution Approach 1:
The patent merges the common processing logic and data retrieval operations into a single shared execution path. User-specific customization is achieved through parameter passing and filtering rather than separate code execution. Multiple user reports can be generated from a single processed dataset, significantly reducing processing usage and memory requirements while maintaining full user customization capability.
Data Source
AI summary
An upload tool includes a network interface, a table generation engine, and an upload engine. The network interface receives a request to generate a data table for a plurality of files. The request includes a username and password for the files, and one or more column definitions. The table generation engine generates the data table that includes the column definitions and a plurality of data rows, each row comprising data from at least one of the plurality of files. The upload engine creates a plurality of batches from the data table and uploads in batch in parallel.