Task Generation System for On-Demand Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face challenges in efficiently processing large existing data sets within an on-demand code execution environment due to limitations in communication speed and computing power, and the lack of unique identifiers for data items, which inhibits the adoption of such systems, especially for new users with pre-organized data sets.
Innovation Solution
A task generation system that transforms data items from a data set into a stream of calls to the on-demand code execution environment, using unique identifiers to ensure each data item is processed, and allows for simultaneous processing of new and backlogged data items by establishing a demarcation point, enabling real-time processing of new items while gradually processing backlogged items.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If an on-demand code execution environment is used to process large existing data sets, then processing capability is improved, but communication speed limitations and lack of unique identifiers prevent effective processing
Solution Approach 1:
The system segments a large existing data set into smaller batches of data items. Each batch is processed separately through the on-demand code execution environment, making the processing manageable despite communication speed limitations. The task generation system divides the overall processing task into multiple smaller tasks that can be executed sequentially or in parallel.
Solution Approach 2:
The system performs preliminary actions by generating unique identifiers for each data item before processing begins. This preliminary step enables the task generation system to track and manage data items effectively, ensuring that each item can be processed and tracked through the communication interface despite speed limitations.
2Productivity
If all data items are processed simultaneously, then processing efficiency is improved, but system limitations prevent handling of large data sets
Solution Approach 1:
The system divides a large data set into smaller manageable batches, allowing the on-demand code execution environment to process each batch without being overwhelmed. This segmentation enables the system to handle large quantities of data by processing them in controlled portions rather than attempting to process everything simultaneously.
Solution Approach 2:
The task generation system establishes a continuous processing workflow where batches of data items are continuously generated, submitted, and processed. This continuous action maintains processing efficiency while managing large data sets through systematic batch handling rather than intermittent processing.
3Measurement precision
If unique identifiers are assigned to all data items, then tracking and processing accuracy is improved, but system complexity increases
Solution Approach 1:
The system implements self-service by automatically generating unique identifiers for each data item through the task generation system. This automated identifier generation improves tracking accuracy without requiring manual intervention or complex external systems. The identifiers are generated as part of the normal data processing workflow, minimizing additional complexity.
4Speed
If the system processes new data items in real-time, then responsiveness is improved, but backlogged data items may be neglected
Solution Approach 1:
The task generation system maintains continuous processing of both new and backlogged data items through an ongoing stream of task calls. Rather than switching between real-time and batch processing, the system continuously generates and submits tasks for all data items, ensuring that new items are processed as they arrive while backlogged items continue to be processed without interruption.
Solution Approach 2:
The system performs preliminary actions by establishing a demarcation point that separates new data items from backlogged items. This preliminary classification enables the system to prioritize and process new items in real-time while systematically working through backlogged items, preventing either category from being neglected.
Data Source
AI summary
Systems and methods are described for transforming a data set within a data source into a series of task calls to an on-demand code execution environment or other distributed code execution environment. Such environments utilize pre-initialized virtual machine instances to enable execution of user-specified code in a rapid manner, without delays typically caused by initialization of the virtual machine instances, and are often used to process data in near-real time, as it is created. However, limitations in computing resources may inhibit a user from utilizing an on-demand code execution environment to simultaneously process a large, existing data set. The present application provides a task generation system that can iteratively retrieve data items from an existing data set and generate corresponding task calls to the on-demand computing environment, while ensuring that at least one task call for each data item within the existing data set is made.