Asynchronous Task Execution Framework with State Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing cloud-based systems face challenges in efficiently executing multiple tasks across services, including determining successful execution, handling resiliency, task ordering, and dynamic configuration, while lacking live tracking and aborting capabilities.
Innovation Solution
An asynchronous task execution framework that utilizes task metadata and a queuing system to publish tasks for execution and track their states, enabling resilient, dynamically retryable, and inter-dependently managed task execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a queuing system with task metadata is used to publish and track tasks, then task execution reliability and progress tracking are improved, but system complexity increases
Solution Approach 1:
A task queue acts as an intermediary component between task producers and task consumers. The queue receives tasks from producers, stores them with metadata, and delivers them to consumers when ready. This mediator pattern isolates the complexity of task coordination from the core business logic, improving reliability while managing system complexity through a dedicated coordination layer.
Solution Approach 2:
The system implements feedback mechanisms by tracking task state changes (pending, running, succeeded, failed) and recording them in the task queue metadata. This allows continuous monitoring of task execution progress and provides information for retry logic and dependency management, enhancing reliability through informed decision-making based on actual task outcomes.
2Reliability
If dynamic retry configurations are implemented, then task execution resilience is improved, but control logic complexity increases
Solution Approach 1:
The system stores retry configuration as dynamic metadata within the task queue, allowing different retry strategies (number of retries, delay intervals, conditions) to be defined per task without hardcoding logic. This enables resilient task execution through configurable parameters rather than complex control logic, as the queue system can interpret and apply the stored retry metadata automatically.
Solution Approach 2:
Retry behavior is controlled by changing parameters stored in task metadata rather than altering control logic structure. The system modifies parameters such as retry count, delay time, and success/failure conditions based on task-specific configuration, allowing resilient execution through parameter variation rather than complex conditional logic.
3Loss of information
If task metadata is recorded in a database table, then progress tracking capability is improved, but data storage requirements increase
Solution Approach 1:
The task queue metadata structure serves multiple functions simultaneously: it stores task identification information, tracks execution state (pending/running/succeeded/failed), records timing information, stores retry configuration, and maintains dependency relationships. By consolidating these diverse information types into a unified metadata structure within the database table, the system achieves comprehensive progress tracking without proportionally increasing storage requirements.
Data Source
AI summary
Methods, systems, and computer-readable storage media for generating, by a task generation service, task metadata representative of a task that is to be executed in the a cloud-based system, recording the task metadata in a table of a database system, publishing the task metadata to a queueing system, receiving, by a task consuming service of a plurality of task consuming services, the task metadata from the queueing system, executing, by the task consuming service, the task, and during execution, updating a state of the task within the table of the database system.