Data Interface for Distributed Computing Task Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed technical computing environments face limitations in managing and handling data associated with distributed tasks, particularly due to reliance on shared file systems that restrict the types of clients and workers that can participate and fail to inform entities of data changes, leading to inefficient computation and reduced interactivity.
Innovation Solution
A system and method that utilize an interface associated with a data repository to programmatically control data operations, including notification of events and maintenance of data integrity, allowing for the transfer of data between computational engines and enabling efficient management of tasks across multiple computing resources, compatible with various data repository types such as file systems, databases, and remote storage arrays.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is stored in a shared file system accessible to computational devices, then data accessibility is improved, but the types of clients and workers that can participate are restricted and data change notification capability is lost
Solution Approach 1:
The patent introduces an interface as an intermediary component between computational devices and the data repository. This interface layer enables diverse clients and workers to access data while providing centralized control for data change notifications, thus resolving the contradiction between versatility and information loss.
Solution Approach 2:
The system segments the data management functionality by separating the data repository from the access interface. The interface is divided into distinct components that handle different operations (reading, writing, notifying), allowing independent optimization of each function while maintaining overall system versatility.
2Productivity
If computation is distributed across multiple computational devices, then computation time is reduced, but data management complexity increases
Solution Approach 1:
The interface is designed as a universal data management layer that serves multiple computational devices simultaneously. It provides multi-functional capabilities including data storage, retrieval, validation, and notification services to all workers and clients, simplifying individual device complexity while enabling distributed computation.
Solution Approach 2:
The interface acts as a mediating layer between multiple computational devices and the underlying data repository. It abstracts the complexity of distributed data management by providing standardized access methods and centralized control mechanisms, allowing computational devices to focus on their processing tasks.
3Reliability
If data validation occurs in the process generating the file, then data integrity is maintained, but the ability to inform entities of data changes is lost
Solution Approach 1:
The system implements feedback mechanisms where the interface monitors data changes and notifies registered entities of these changes. Data validation ensures integrity while the feedback loop through notifications preserves information about data change events, resolving the contradiction between reliability and information loss.
Solution Approach 2:
The interface serves as an intermediary that both validates data (maintaining integrity) and tracks data changes (preserving information). This dual role allows the system to simultaneously ensure data reliability while informing entities of changes through notification events.
Data Source
AI summary
A method and system is disclosed for providing a distributed technical computing environment for distributing technical computing tasks from a technical computing client to technical computing workers for execution of the tasks on one or more computers systems. Tasks can be defined on a technical computing client, and the tasks organized into jobs. The technical computing client can directly distribute tasks to one or more technical computing workers. Furthermore, the technical computing client can submit tasks, or jobs comprising tasks, to an automatic task distribution mechanism that distributes the tasks automatically to one or more technical computing workers providing technical computing services. The technical computing worker performs technical computing of tasks and the results of the execution of tasks may be provided to the technical computing client. Data associated with the tasks is managed by a programmable interface associated with a data storage repository. The interface allows the various entities of the distributed technical computing environment to access data services performable by the interface or by a file system or a database and database management system associated with the data.