Large File Upload Workflow Using Chunking and User Entitlements
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing application programming interfaces (APIs) for file upload are limited by file size, often supporting only up to 10 MB, leading to frequent failures and requiring labor-intensive reintegration for new users, and traditional hard-coding methods are inefficient for managing user-dependent upload processes.
Innovation Solution
A system and method for large file upload that includes chunking files into smaller sizes, processing them in a distributed computing environment, and managing upload procedures based on user entitlements, with event history tracking and configurable notification protocols.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional API integration methods are used for file upload, then the system can support file uploads, but the file upload size is limited to 10 MB and failures occur frequently
Solution Approach 1:
The patent divides large files into smaller chunks (e.g., 10 MB segments) before upload. Each chunk is uploaded separately through the API, then reassembled on the server side. This segmentation allows the system to handle files of any size while maintaining compatibility with API limitations, thereby improving both reliability and adaptability.
Solution Approach 2:
The patent introduces an intermediary file upload service that acts as a mediator between the user and the document management system. This service manages the chunking, upload, and reassembly process, absorbing the complexity and preventing direct API failures from affecting the core system. The intermediary layer improves reliability by handling errors gracefully and maintaining upload state information.
2Adaptability or versatility
If hard-coding methods are used to plan upload processes based on user roles, then the upload process can be customized, but the system requires continuous reintegration and recoding when new users are added
Solution Approach 1:
The patent implements dynamic configuration where upload processes are determined by user entitlements and roles stored in the database. When a user uploads a file, the system queries the user's entitlements and automatically adjusts the upload process accordingly. This dynamic approach eliminates the need for hard-coding and reintegration when new users or roles are added, as the system adapts automatically based on stored configuration data.
Solution Approach 2:
The patent changes the approach from fixed hard-coded processes to parameter-driven processes. User entitlements, roles, and upload configurations are stored as modifiable parameters in the database. This allows administrators to configure upload behaviors for different user types without modifying the codebase, reducing integration complexity while maintaining adaptability.
3Productivity
If existing API integration methods are used, then file upload can be performed, but fault points lead to upload failures downstream
Solution Approach 1:
The patent implements beforehand cushioning by pre-uploading files to a staging area database before final processing. This intermediate storage acts as a buffer, allowing the system to recover from downstream failures without losing the uploaded data. The event history tracking also provides early warning of potential issues, enabling preventive actions before failures occur.
Solution Approach 2:
The patent implements comprehensive feedback mechanisms through event history tracking that monitors each stage of the upload process. The system tracks events from file reception through processing to final storage, providing feedback loops that enable automatic error detection, recovery, and notification. This feedback system improves reliability by identifying and correcting issues before they propagate through the system.
Data Source
AI summary
Systems and methods for large file upload, configurable with user workflows are disclosed. An upload service can receive a data file including metadata associated with a user, then upload the data file to a first database within a distributed computing environment. The upload service may identify an entitlement associated with the user based on the metadata and upload the data file to a second database. The upload service may determine a notification protocol based in part on the entitlement associated with the user, then generate one or more notifications based on the notification protocol. Each action performed by the upload service may be recorded and used to determine an event history of the data file. The upload service can store the event history.