Shared Memory Data Exchange for Distributed Task Routines
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing complexity and size of data in scientific and technical analyses require advanced mechanisms for organization, reproducibility, and accountability, as well as efficient allocation and dynamic re-allocation of computing resources to support distributed task execution and data sharing.
Innovation Solution
A system that includes a processor and storage to manage job flows by retrieving and executing task routines from federated areas based on data dependencies, dynamically allocating task pods, and using message queues to coordinate task execution across multiple container environments, ensuring efficient resource utilization and error handling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data objects are exchanged between task routines using traditional inter-process communication methods, then task execution can be coordinated, but communication overhead and execution time increase significantly
Solution Approach 1:
The patent merges the memory spaces of multiple task routines into a single shared memory space, allowing task routines to directly access and exchange data objects without traditional inter-process communication overhead. This combining of memory resources eliminates the time-consuming data copying and communication protocols associated with conventional methods.
Solution Approach 2:
The patent introduces a memory management routine as an intermediary that handles the allocation, deallocation, and coordination of shared memory space. This mediator manages access to the shared memory, ensuring proper synchronization and data integrity while enabling efficient direct access for task routines.
2Productivity
If computing resources are statically allocated to task routines, then resource management is simple, but resource utilization efficiency decreases during dynamic workloads
Solution Approach 1:
The patent implements dynamic resource allocation where the system automatically adjusts the allocation of computing resources (processors, memory, storage) to task routines based on real-time workload demands. The resource management routine monitors system state and dynamically reconfigures resource distribution, enabling high utilization during peak loads while maintaining simplicity through automated management.
3Reliability
If task routines execute in isolated container environments, then system stability and error containment improve, but inter-task data sharing and coordination become more complex
Solution Approach 1:
The patent creates a universal shared memory space that serves multiple functions simultaneously: it enables data sharing between isolated task routines, provides synchronization mechanisms, manages memory allocation, and maintains system stability. This multi-functional shared memory infrastructure reduces the need for separate communication channels and coordination protocols.
Solution Approach 2:
The memory management routine acts as an intermediary that bridges isolated container environments while maintaining their security boundaries. It handles all data exchange operations between containers, providing controlled access and coordination without requiring complex peer-to-peer communication mechanisms between isolated task routines.
Data Source
AI summary
An apparatus includes a processor to: based on data dependencies specified in a job flow definition, identify first and second tasks of the corresponding job flow to be performed sequentially, wherein the first task outputs a data object used as an input to the second; store, within a task queue, at least one message conveying at least an identifier of the first task, and an indication that the data object is to be exchanged through a shared memory space; within a task container, in response to storage of the at least one message within the task queue, sequentially execute first and second task routines to sequentially perform the first and second tasks, respectively, and instantiate the shared memory space to be accessible to the first and second task routines during their executions; and upon completion of the job flow, transmit an indication of completion to another device via a network.