Stage File Objects for Parallel Data Chunking in Cloud Integration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud-based integration services face challenges in processing large amounts of data efficiently, particularly when integrating applications, as standard external calls are insufficient for complex processing tasks involving hundreds of thousands of records, leading to significant time delays.
Innovation Solution
The implementation of a cloud-based integration service that utilizes stage file objects within a visual development tool, allowing for parallel processing through chunking, enabling developers to visualize and configure stage file actions such as read, write, and zip operations, which break down large files into manageable chunks for simultaneous processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If standard external calls are used to transfer data between applications, then the integration process is simple to implement, but the processing time becomes enormous when dealing with hundreds of thousands of records
Solution Approach 1:
The patent divides large files into smaller chunks that can be processed in parallel. The stage file object breaks down a large file into multiple smaller segments, allowing each segment to be processed independently and simultaneously by different processing objects, thereby reducing overall processing time while maintaining integration capabilities
Solution Approach 2:
The patent performs preliminary actions by staging files before actual processing occurs. The stage file object prepares data in advance by breaking it into chunks and making it available for parallel processing, so that when processing begins, multiple operations can occur simultaneously rather than sequentially
2Adaptability or versatility
If all data is transferred from one application to another, then the second application has complete access to the data, but the data transfer time and resource consumption increase significantly
Solution Approach 1:
The patent extracts only the necessary portions of data from the source application through stage file objects. Instead of transferring entire files or all data, the stage file object extracts specific chunks that are then processed and made available to the target application, reducing transfer time while maintaining data accessibility
Solution Approach 2:
The patent introduces stage file objects as intermediary components between source and target applications. These intermediaries hold staged versions of files that can be accessed by multiple processing objects without requiring direct transfer from the source, thereby reducing transfer time while maintaining data accessibility through the intermediary stage
Data Source
AI summary
Described herein are systems and methods for providing stage file objects in a visual design tool for integration development. The integration can be developed on a cloud based integration service that can receive, via a visual development tool interface, selection of a stage file object for insertion into an integration flow. The cloud based integration service can also receive a selection of a type value for the stage file object. The scope for the stage file object can be determined, and the stage file object can be displayed in the integration flow based on the type value and the scope.