Distributed Job Manager for Microservices Data Locality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems for managing distributed computing jobs face performance bottlenecks, single points of failure, and inefficiencies due to shared storage configurations, which result in slower performance, data synchronization issues, and increased network traffic.
Innovation Solution
A distributed system architecture that employs a metadata server for file distribution and a job server for coordinating data access across compute nodes, utilizing a scheduler extender to prioritize node selection based on data locality and resource availability, enabling hyperconvergence without modifying the standard scheduler.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If data is stored in a shared siloed infrastructure accessed remotely by all jobs, then data centralization is achieved, but network traffic increases and performance decreases
Solution Approach 1:
The patent segments the monolithic shared storage infrastructure into distributed file systems across multiple nodes. Each node maintains local storage resources, eliminating the single centralized storage bottleneck and enabling parallel access without network traffic congestion.
Solution Approach 2:
The patent introduces a new dimensional approach by implementing a hierarchical storage architecture where data can be accessed both locally on compute nodes and remotely through the distributed file system, adding spatial diversity to data access paths and reducing network dependency.
2Reliability
If a lock manager is used to maintain data synchronization across shared storage, then data consistency is improved, but system complexity and performance bottlenecks increase
Solution Approach 1:
The patent extracts the lock manager functionality from the core system architecture, implementing it as an optional plugin or extension. This allows data synchronization to be achieved when needed without imposing the complexity of a centralized locking mechanism on the entire distributed file system architecture.
Solution Approach 2:
The patent introduces an intermediary layer between compute nodes and storage resources that handles synchronization requests. This mediator manages data consistency through coordinated access protocols without requiring a centralized lock manager, reducing system complexity while maintaining reliability.
3Adaptability or versatility
If software containers include all necessary software resources, then interoperability across different computing environments is improved, but container size and deployment complexity increase
Solution Approach 1:
The patent implements a universal base container image that includes common software resources and dependencies. This multi-functional base image can serve multiple different computing environments and job types, reducing the need for separate specialized containers while maintaining adaptability across platforms.
Solution Approach 2:
The patent performs preliminary action by pre-configuring container images with commonly needed software resources, dependencies, and runtime environments during the build stage. This advance preparation reduces deployment complexity and ensures interoperability without requiring complex runtime configuration or resource allocation.
4Stability of the object's composition
If data is accessed remotely from shared storage, then data centralization is maintained, but access latency increases
Solution Approach 1:
The patent implements local quality by enabling compute nodes to cache frequently accessed data locally or access it through high-speed interconnects. This creates quality variations in data access paths, with hot data accessed rapidly from local caches and cold data accessed from centralized storage, optimizing overall system performance.
Solution Approach 2:
The patent performs preliminary action by pre-loading or caching data that is likely to be accessed by upcoming jobs. This anticipatory data preparation reduces access latency when jobs need to retrieve data, as the information is already positioned in optimal locations before actual access occurs.
Data Source
AI summary
Two servers implemented as containerized applications may manage the storage of data and the access of that data by compute jobs in a distributed system. A metadata server may distribute data on ingress and assign files to particular storage volumes. The metadata server may then provide a lookup function for files and be configured to distribute a file to other volumes when necessary. A job server may launch jobs as containerized applications and coordinate data access across jobs.